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Introduction 



Application review is the most important part of the Department of Education's funding 
process and is designed to ensure that projects are worthwhile and that awards are fair 
and impartial. The critical step in application review is competitive, objective evaluation 
by reviewers like yourself, who are carefully chosen for their expertise and ability to 
assess a proposed project’s quality. Because your reviews are the foundation for higher- 
level decisions, the Department has developed this workbook to clarify your 
responsibilities and sharpen your review skills. The workbook is intended both to make 
your task easier and to improve the quality of application reviews. 



Workbook Objectives 

When you have completed the workbook, you will be able to: 

1) Understand the application review process and your role 
and responsibilities as a reviewer; 

2) Comprehend the roles of other personnel; 

3) Read and analyze applications more efficiently; 

4) Apply selection criteria in reviewing an application; 

5) Write detailed comments about an application's strengths 
and weaknesses; 

6) Score applications correctly; and 

7) Serve on a panel. 



Workbook Contents 

Used in its entirety, the workbook provides a comprehensive orientation to application 
review. Although the Department urges all reviewers to complete or at least read 
through the whole workbook, you may prefer to focus on the sections most relevant to 
your own needs. If you have never reviewed applications before, the workbook offers 
general information and skill-building exercises. If you are an experienced reviewer, you 
can use the workbook to answer specific questions, refresh your memory, or refine your 
skills. The table of contents will direct you to the information you need. 



The workbook contains three modules: 



Module I: The Process of Grant Application Review and 

the Key Personnel Involved; 



Module II: Understanding Selection Criteria; 

Module III: Writing Comments, Scoring, and Team Skills. 



Module I explains the purpose of the review process, provides an overview of the 
process, and describes the roles of personnel directly involved in the process. 



Module II focuses on the selection criteria used in evaluating applications and the 
importance of using good reading and analytical skills. Tips for both skills are provided. 
The discussion of each criterion is followed by an exercise designed to give you practice 
in assessing an application's response to that criterion. 

Module III describes the Technical Review Form that you will use and discusses how to 
write comments, score applications, and participate in panel review. The latter three 
sections include practice exercises. A glossary of key terms is also provided at the end of 
the workbook. Terms included in the glossary are printed in bold type the first time they 
appear in the workbook. 



Using the Workbook 

Because the workbook is designed to be self-instructional, you may not need to complete 
all of the exercises provided. The workbook is meant to be a practical tool. Reviewers 
who complete the workbook as part of a formal orientation session, however, will be 
expected to complete exercises as directed by the orientation trainer. 

Please use this workbook and keep it as a general reference guide to application review. 
Whether you use it on your own or as part of a Department orientation session, you 
should feel free to make notes in the margins. If you have questions that are not 
answered by the workbook, do not hesitate to ask your program office contact or 
orientation trainer for clarification. Also, the workbook may be supplemented with 
program-specific guidelines and instructions. Be sure to read these as well before 
beginning to review applications. 




iv 



7 



MODULE I: 



The Process of Grant Application Review 
and the Key Personnel Involved 



MODULE I: The Purpose of the Review Process 



A. The Purpose of the Review Process 



Background 

The Department of Education was established in May of 1980 to 
consolidate Federal education programs under a single agency. The 
Department's purpose is to promote national education policy as 
determined by Congress, while enabling State and local governments to 
exercise their constitutional right to set specific educational objectives. In 
accordance with Federal legislation, the Secretary of Education 
determines the general educational priorities for each fiscal year, and 
these priorities are monitored by a framework of program offices within 
the Department. Program offices are empowered to award Federal 
monies set aside by Congress to fund projects that further legislated 
educational objectives. 



Purpose 

Program offices are responsible for identifying projects that best address 
the specific educational needs targeted by legislation. To accomplish this 
task, they rely on a process of review that permits all eligible institutions 
to submit applications and compete for available funding. A certain 
percentage of such funding is awarded in the form of discretionary grants 
and cooperative agreements. For the purposes of this workbook, 
discretionary grants and cooperative agreements will be referred to 
collectively as grants because they are reviewed in the same manner. The 
difference between the two funding modes does not affect your role as a 
reviewer. 

The application review process combines an initial, objective evaluation 
by reviewers with internal evaluation by program staff to arrive at a 
comprehensive and impartial assessment of a proposed project's 
likelihood of furthering legislated educational priorities. Outside review is 
based on the Education Department General Administrative Regulations 
(EDGAR) and any specific program regulations and provides program 
offices with ranked applications. The review process ensures that funded 
projects not only serve the purposes outlined by national education policy, 
but are best qualified to do so. 
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MODULE I: An Overview of the Review Process 



B. An Overview of the Review Process 



The diagram to your left illustrates the 14 steps of the review process. 
Each step in the diagram is numbered to correspond to the steps fisted in 
the discussion below. 

Step 1 

Discretionary grants and cooperative agreements have their genesis in 
Congressional education acts, as do all funding methods used by the 
Department. 

Step 2 

The Department establishes regulations that govern how each act's 
objectives are to be carried out. 

Step 3 

Congress appropriates funds, and the Department conducts a process 
designed to award those funds. 

Step 4 

The Department establishes a schedule of awards and approves an 
application package for each funding program. 

Step 5 

A notice that the program is accepting applications is published in the 
Federal Register , listing submission deadlines as well as application 
requirements. The appropriate program office is also listed and 
undertakes responsibility for printing and mailing out application 
packages as they are requested. Application packages include a letter that 
outlines the legislative purposes that must be addressed by the proposed 
project, a list of the criteria that will be used in evaluating an application 
ana how they will be scored, and the necessary application forms. 
(Because applicants draft their applications in conformance with the 
selection criteria, your understanding of and attention to those same 
criteria are essential.) 

Step 6 

Completed applications are sent to the Department, where their receipt is 
officially logged by the Department's Application Control Center (ACC). 
The ACC returns to applicants any applications that have been rejected 
because they were submitted after the published deadline. 
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Step 7 

The program office screens timely applications for eligibility and 
conformity to the general administrative regulations governing 
Department programs and selects application reviewers. 

Step 8 

Applications enter the evaluation phase and are assessed by reviewers like 
yourself. Reviewers' assessments are entered onto a Technical Review 
Form. Some applications may only be read by individual reviewers; their 
reviews are collected by the program office. In other cases, reviewers will 
convene as a panel to discuss their reviews. 

Step 9 

The program office receives reviewers' written evaluations, verifies their 
adequacy, and averages the points assigned by reviewers to establish an 
average for each application as well as a rank order of all applications. 
The Department's Grants and Contracts Service (GCS) office has a 
standardized-score software package that can be used to make statistical 
adjustments to compensate for the various scoring patterns among 
reviewers. If the program office wishes, GCS can use this package to rank 
the applications. 

Step 10 

The program office reviews the applications again, in order of rank, and 
prepares written funding recommendations. 

Step 11 

The Assistant Secretary of the program office reviews the program office 
recommendations, checks the adequacy of documentation supporting the 
recommendations, and approves a final list, or slate, of recommended 
projects and funding amounts. 

Step 12 

GCS receives the list of recommended projects and negotiates the final 
funding level for each. GCS then notifies the Office of Legislation, which 
notifies Congress and successful applicants of the funds awarded and 
issues the awards. After all awards have been issued, the program office 
notifies applicants whose applications have been rejected. 

Step 13 

The program office distributes copies of reviewers' evaluations to 
applicants who request them. EDGAR regulations entitle all applicants 
to receive such copies, but although a list of all the reviewers' names is 
furnished, evaluations are not attributed to specific reviewers. 
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MODULE I: An Overview of the Review Process 



Step 14 

The program office and GCS monitor awarded grants until they are 
completed and then close them out. 



The review process is carefully designed to build to a decision, allowing a 
maximum of input that is both expanded and refined at each successive 
level. Application review is the essential step in the process. Although 
the number of review stages should ensure that each application is 
accurately evaluated, your review has a major impact on final funding 
recommendations. 
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MODULE I: A Description of the Roles of Personnel Directly Involved in the Review 
Process 



C. A Description of the Roles of Personnel Directly Involved in the Review Process 



As the previous section notes, the review process involves a number of 
personnel. Reviewers may not work directly with these personnel, but 
each one is involved at some point with reviewers' evaluations. A better 
understanding of their roles, therefore, may help you to define your own 
responsibilities. 



KEY PERSONNEL 

■ Grants Officer 

■ Program Officer 

■ Panel Manager/Chairperson 

■ Reviewer 



Grants Officer 

Field reviewers who are not asked to review applications onsite will have 
little or no contact with Department personnel, except by correspondence 
or telephone. These reviewers will deal directly with program office staff, 
who will answer their questions and refer them to other Department 
personnel if necessary. Maiw reviewers, however, will be asked to serve at 
a designated review site and will participate in a training and orientation 
session. A Grants Officer from the Grants and Contracts Service office, 
which supervises the negotiation and administration of Department 
grants, will participate in this orientation, conducted by the program staff. 

The role of the Grants Officer is to interpret grants management policies 
to reviewers/panelists and to resolve any conflicts that may arise over 
questions of policy. The Grants Officer does not provide input into 
application review, but is involved as an adviser. If present during a panel 
discussion, the Grants Officer may be called upon to explain, as 
appropriate, the key provisions of EDGAR as well as the issues of 
confidentiality and conflict of interest. 

The Grants Officer is a major information resource for reviewers. During 
the final stage of grant negotiation, the Grants Officer will review your 
comments to assist in negotiating funding levels and any substantive 
changes needed in the proposed project. 
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Process 

Program Officer 

Whether you read applications in Washington, D.C., or in the field, you 
will interact with a Program Officer. This individual is a member of the 
program's professional staff whose role is to advise reviewers about 
program regulations and to explain the following: 

■ Review process; 

■ Selection criteria; 

b Reviewers' responsibilities; and 
b Technical Review Forms. 

While you may serve on a panel of reviewers, Program Officers do not 
serve as panelists but are present as facilitators of the process. They are 
not allowed to comment on applications, discuss the prior history of the 
applicant, or divulge their personal opinion or analysis. They will, 
however, mediate the panel review to ensure that each application is 
thoroughly and fairly assessed. They will also check all Technical Review 
Forms for adequacy and accuracy and request additional documentation, 
if necessary, before the panel is dismissed. After reviews are completed, 
Program Officers use reviewers' comments to prepare a summary 
recommendation for each application. 

Panel Manager/Chairperson 

Reviewers who serve as panelists will also encounter a Panel Manager or 
Chairperson, although this role is often undertaken by a Program Officer. 
Panel Managers do not participate in panel review, except to stress fair 
review. Panel Managers are responsible for panel logistics, distributing 
supplies and applications, monitoring the progress of individual reviews, 
and fostering panel discussion of the application. They can also answer 
procedural questions and direct reviewers to appropriate personnel 
regarding travel and other administrative matters. 

Reviewer 

Reviewers are selected by program offices on the basis of their general 
and specialized experience in a program area. Reviewers are expected to 
draw upon their expertise in evaluating applications according to the 
selection criteria announced by the program office. The review process 
involves many stages of sifting and weighing, but the reviewer is the 
primary source of objective assessment and bears a large responsibility for 
making accurate evaluations. Reviewers provide input that is different 
than the analysis made by program officers. 

The purpose of your review is to provide a specific overall 
recommendation that is supported by documentation drawn directly from 
the application. You must always be conscious that your evaluations will 
form the basis of recommendations made at higher levels of review. 
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MODULE I: A Description of the Roles of Personnel Directly Involved in the Review 
Process 

Because your evaluations may also be made available to applicants upon 
request at the close of the review process, you should be careful to avoid 
disparaging language in your comments and focus instead on constructive 
criticism that could benefit denied applicants who may wish to reapply at a 
later date. 

Any personal knowledge you might have about a particular applicant 
should be suppressed in the interest of objectivity, and any personal 
opinions or feelings about a proposed project should be actively ignored 
unless they relate directly to the announced selection criteria. Similarly, 
you must base vour evaluations solely on information provided in the 
application, without assuming greater knowledge on the part of the 
applicant or filling in gaps from your own personal knowledge. Under no 
circumstances will you contact an applicant for clarifying information . 

Panel Discussion: A panel discussion may supplement your individual 
review. The program office may determine the need for panel discussion 
in advance because of the importance or magnitude of the award being 
offered, or it may decide upon panel discussion during the evaluation 
process if reviewers’ individual evaluations exhibit a wide disparity of 
scores. Panel discussion usually involves three or more application 
reviewers and is supervised by a Panel Manager who facilitates the 
process. 

The purpose of panel discussion is to allow for a thorough discussion of an 
application. If asked to serve on a panel, you must be prepared to 
substantiate your written evaluations orally and to discuss applications 
objectively. Individual ratings are discussed in turn so that all panel 
reviewers have the opportunity to reconsider their evaluations. General 
consensus is not, however, the objective; discussion is intended to 
stimulate the thinking of the panel by allowing different viewpoints to 
interact and generate deeper understanding. Panel discussion is only 
meant to ensure that an application's particular strengths and weaknesses 
have been duly noted by the reviewers and that reviewers are satisfied 
with their final evaluations. Reviewers who wish to change their original 
evaluations after consideration may do so, as long as they provide a reason 
for the change. 

Confidentiality: At no time when reviewing applications, or after, will you 
discuss the applications, comments, recommendations, evaluations, review 
scores, names of applicants, or names of other reviewers with anyone not 
involved in the review process. Even discussion with other panelists is 
proscribed outside the panel meeting room. Applications are submitted to 
the Department in confidence, and you must respect that confidentiality. 
Reviewers' comments enjoy a similar confidentiality and review forms are 
never distributed to applicants before reviewers' names are deleted. A list 
of reviewers' names may be furnished, however, if requested. 
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CONFIDENTIALITY 

■ Do not discuss applications with others. 

■ Do not remove applications from the review location. 



Conflict of Interest: The Department also requires reviewers to notify the 
Department of any finanical interest they have in any application in the 
competition. Program offices screen reviewers and applications before distribution 
to ensure that no pending application involves a reviewer; a reviewer’s spouse; 
a profit or nonprofit organization for which the reviewer serves as an officer, 
director, trustee, partner, or employee; or any person or organization with whom 
the reviewer has a present or anticipated relationship involving employment or 
financial interest. Similarly, no panel is allowed to involve two reviewers from 
the same organization or institution. 

Despite Department precautions, a conflict of interest may nevertheless occur. 
If you feel that you have a financial interest in any application in the competition 
you are responsible for reporting that interest to program office staff. This 
responsibility is strictly enjoined upon you; an undetected conflict of interest 
undermines the entire review process. 



CONFLICT OF INTEREST 

You must report a finanical interest to program staff if any 
application in the competition involves: 

1) Yourself; 

2) Your spouse; 

3) An organization for which you serve as an officer, 
director, trustee, partner, or employee; 

4) A person or organization with which you have 
present or anticipated employment or financial 
interest. 
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MODULE 1: Conclusion 



The following table summarizes your responsibilities as a reviewer. 



1) To provide a specific and well-documented critique 
of each application. 

2) To prepare constructive written comments, instead 
of offering snide commentary. 

3) To evaluate applications independently of personal 
feelings or knowledge about the applicant. 

4) To participate in panel discussion, if required. 

5) To maintain confidentiality. 

6) To report any conflict of interest. 



This module is designed to acquaint you with the broad outline of the 
review process and your own role. That role is, in general, to provide a 
cogent analysis of applications that will enable program office staff to fund 
projects that best address legislated educational priorities. In specific, you 
will be asked to comment upon and score each application on a Techmcal 
Review Form, according to a given set of selection criteria. 

A sample illustration of a reviewer's work appears on the page to your 
right. Here, the reviewer has commented upon and scored an 
application's response to the Plan of Operation selection criterion. As the 
sample indicates, you are responsible for analyzing the strengths and 
weaknesses of an applicant's response to each selection criterion and 
assigning a numerical score that corresponds to the adequacy of that 
response. These scores and comments will be used by program office staff 
to rank the applications. 

Therefore, although review involves your personal expertise, it must also 
be firmly anchored to the selection criteria listed on the Technical Review 
Form. Most of the criteria you will encounter are explained in the next 
module and are accompanied by practice exercises. 



REVIEWER RESPONSIBILITIES 



Conclusion 
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SAMPLE OF REVIEWER’S WORK ON 
A TECHNICAL REVIEW FORM 



b) PLAN OF OPERATION Maximum 20 points Awarded 9 

The application has a plan that: 

- Shows quality 

- Shows design 

- Ensures proper and effective administration 

- Describes how objectives relate to purpose 

- Presents plans for use of resources and personnel 

- Adheres to EEO regulations 

Strengths : 

The applicant proposes three measurable goals: to decrease the tribal 
unemployment rate, to place 85 percent of project graduates into jobs, and 
to increase the number of tribe members in off-reservation jobs (11-16, A1-3). 
The first four project objectives detail the number Emanons to be trained in 
each of the project’s vocational education areas and provide job projec- 
tions (Attachment C(2)) to support the training objectives (11-16, B1 -4). Ac- 
cording to the documentation provided, the objectives seem adequate to 
accomplish the project’s goals. In addition, the plan of operation is design- 
ed to ensure that training activities are tailored to the availability of jobs in 
the area (11-16, B1-4; 11-17, C2; 11-18, C4). In two of the vocational education 
areas targeted by the project-electronics assembly and tourism- the appli- 
cant has already arranged for trainee placement (11-17, C2; 11-18, C4). 

Weaknesses : 

The plan of operation proposes four goals. The first three-decreasing unemploy- 
ment, job placement, and increasing off-reservation employment-are measurable. 
The fourth goal, however-increasing Emanon self-esteem-is not measurable as 
stated and the plan of operation suggests no procedure for assessing and monitoring 
the success of this goal (11-16, A). Since the overall objective of the project is to 
increase Emanon economic independence, some procedure for measuring self- 
esteem would be essential for determining whether the project promotes attitudes of 
self-sufficiency. A second weakness is the absence of any plan to assess student’s 
basic skill levels (11-17, C). Since the project concentrates its efforts in vocational 
education, its success will depend on how well students are trained. If instruction is 
not conducted at a skill level appropriate to the students, the trainees will not be 
equipped to compete for jobs. Third, the time line proposed for accomplishing 
project activities is not comprehensive or detailed. The number of classroom hours 
to be offered is noted(ll-17, C), but the hours are not justified by any documenta- 
tion indicating that they are sufficient for the objectives involved. Courses are 
scheduled to finish by the time job opportunities will be available (11-19, F), but no 
detailed master schedule is provided for all activities-recruitment, startup, teacher 
training, instruction, monitoring, evaluation, etc. A comprehensive management 
plan is not evident. 
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MODULE II: 



Understanding Selection Criteria 



Introduction 



This module is divided into three sections. The first two, Reading Applications and 
Analysis of Applications, provide pointers on the physical and mental process of reviewing 
an application. The third section, Selection Criteria, introduces the Education Department 
General Administrative Regulations (EDGAR) and includes a subsection on each of the 
EDGAR selection criteria. Each subsection explains what you should look for when 
reviewing an application's response to the criterion and provides a practice exercise. The 
practice exercises are taken from a fictitious application. Although this application is 
representative of those you will encounter as a reviewer, it should not be used as a standard 
against which to evaluate other applications. 

The purpose of this module is to develop your ability to read and analyze an application. 
Therefore, the primary emphasis is on the practice exercises. Each practice exercise begins 
with a passage that responds to the selection criterion being examined. You must read the 
passage and analyze the strengths and weaknesses of the application's response. Space is 
provided after the passage for you to list the strengths and weaknesses that you noted while 
reading. You should list as many strengths and weaknesses as you can, using the review 
guidelines provided for that criterion. If you can, cite the pages where the strengths and 
weaknesses appear. For ease of reference, you may wish to remove the selection criteria 
overview provided at the end of the module and keep it in front of you as you read. The 
overview lists the review guidelines for each criterion. 

Each practice exercise is followed by a master list of strengths and weaknesses. You should 
review the master list to determine whether you are focusing correctly on the requirements 
of that selection criterion. You may want to reread the passage if certain strengths and 
weaknesses on the master list were not apparent to you. The master list is not exhaustive, 
however; you may have noted other strengths and weaknesses. That is why your role as a 
reviewer is so important. 

Once you have reviewed the master list for an exercise, you can move on to the exercise 
provided for the next selection criterion. If the exercises seem difficult to complete, you 
should review the sections on reading and analysis for tips on improving your performance. 
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MODULE II: Reading Applications 



A. Reading Applications 



Reading an application is an active and selective process. You are reading 
with a purpose: to extract information from the text. Your goal is to 

comprehend the text, i.e., to seek information that answers specific questions. 
As a reviewer, your main question is: Does the application address the 
selection criteria? Each one of the selection criteria involves a subset of 
questions that you must ask in order to determine how the application 
responds to that criterion. 

Before you begin your review, you should study the guidelines for reviewing 
the selection criteria to ensure that you have the right questions in mind. 
Then, as you read, focus your attention on locating information that relates 
to those guidelines. This will enable you to slam through nonessential 
information, increasing both the speed and the efficiency of your reading. 
You do not need to give equal time to every word; concentrate on key words 
and phrases that announce relevant information. 

Remember to read critically. Trying to memorize points as you go will 
disrupt your comprehension of the text as a whole. If you think you may 
forget an important point, jot it down on the Technical Review Form or a 
piece of paper as you go. You can also use paper clips to mark important 
passages. You should never write on the application itself, however, since 
copies may need to be returned to the applicant. 

If you have personal concerns or worries, attempt to clear them from your 
mind so that you can bring your full attention to bear upon the application. 
Try to minimize external distractions, too: fatigue, hunger, physical 

discomfort, noise. These factors can impair your ability to concentrate. 



MODULE II: Reading Applications 



READING TIPS 

1) Read with a purpose. 

2) Focus on information related to the criteria. 

3) Skim over nonessential information. 

4) Concentrate on key words/phrases. 

5) Read critically. 

6) Read quickly. 

7) Don't try to memorize. 

8) Take notes, if necessary. 

9) Relax and clear your mind of personal concerns. 

10) Minimize distractions. 



Before you start your review, you should consider the number of applications 
you must review and the amount of time allotted for your review. Then 
decide how much time you can spend on each application. Some 
applications may require more time, others less; but establishing a maximum 
review time will ensure that each application receives sufficient attention. 
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B. Analysis of Applications 



As noted, reviewing an application requires you to look for information as 
you read-information that answers specific questions. When -you find that 
information, however, you must also evaluate it. 

Every application will respond to the selection criteria to some degree; your 
task is first to locate the relevant response and then to analyze the quality of 
that response. Your analysis should be an objective appraisal that focuses on 
how well the application responds to the selection criteria. 



GUIDELINES FOR ANALYSIS 

1) The applicant's intentions must be clear and specific, 
not obscured by meaningless jargon. 

2) The ideas presented must flow logically. 

3) The application must provide a complete response to the 
selection criteria. 

4) The activities outlined in the different sections of the 
application must be consistent with each other. 

5) The activities proposed by the applicant must be consistent 
with current, accepted knowledge and ideas in that field. 
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C. Selection Criteria 



You must read each application and analyze it against a set of announced 
selection criteria. Each of the criteria must be addressed by the applicant 
when the application is being prepared. 

Programs that have their own published regulations will have a specific set of 
criteria to be used in the review process. If a program has not published its 
own regulations, then the selection criteria to be used in reviewing 
applications are those prescribed by EDGAR. There are seven EDGAR 
criteria. 



EDGAR SELECTION CRITERIA 

1) Meeting the Purposes of the Authorizing Statute; 

2) Extent of Need for the Project; 

3) Plan of Operation; 

4) Quality of Key Personnel; 

5) Budget and Cost Effectiveness; 

6) Evaluation Plan; and 

7) Adequacy of Resources. 



These seven criteria will be used throughout the remainder of this workbook. 
Once you understand them, you will know what to look for as you read an 
application and how to evaluate the information that is provided. 

Keep in mind, however, that the program you are reading for may have 
criteria that differ somewhat from the EDGAR criteria. The program staff 
will provide you with their appropriate criteria before you begin reading 
applications. In many cases, the program criteria will address the same issues 
as the seven generic EDGAR criteria listed above, though the wording will 
be different. In some cases, the EDGAR criteria will be supplemented by 
additional criteria that address unique aspects of the particular program. 

Regardless of the specific wording of the criteria, however, the process of 
reviewing the application, analyzing it, and identifying its strengths and 
weaknesses is the same. Thus, once you have learned how to analyze the 
seven EDGAR criteria in this workbook, you will be able to analyze any 
other program-specific criteria. 
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Each selection criterion is weighted with a point value. EDGAR 
recommends point values, but each program office has the authority to assign 
alternative point values through its program-specific regulations. Thus, while 
EDGAR weights Extent of Need for the Project at 20 points, a program 
office may decide that Extent of Need is only worth 15 points and assign the 
additional 5 points to another criterion that it deems more important. The 
flexibility of point values allows program offices to tailor application review 
to their current priorities. The Secretary of Education may also assign 
additional points to an application that meets his own educational priorities 
as long as the basis for awarding these extra points is noted in the original 
application announcement. Thus, the purpose of the selection criteria is to 
ensure that all applications meet certain established standards, while 
permitting variable point values that will accommodate shifts in educational 
policy and objectives. 

As a reviewer, you are responsible for assessing an application's response to 
the substantive requirements of each criterion and scoring that response as 
indicated by the point values noted on the Technical Review Form. Since 
point values may vary, and program offices may also establish additional 
selection criteria, you must be sure to study all application review guidelines 
provided by the program office. 

You will find a discussion of each general selection criterion in the sections 
that follow, as well as practice exercises designed to enhance your reading 
and analysis skills. An overview at the end of the module recapitulates the 
review guidelines for each selection criterion. You can remove this overview 
from the workbook for quick reference while completing the practice 
exercises. 
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Criterion: Meeting the Purposes of the Authorizing Statute 



Every program is authorized by a statute, and grants are awarded to projects 
that best carry out the purposes designated in that statute. Applicants should 
be aware of the statute's purposes, and their applications must demonstrate 
how their projects will serve those purposes. 



When you review an application's response to this criterion, you must ask: 

1) What are the purposes of the authorizing statute? 

2) What are the objectives of this project? 

3) How will these objectives further the purposes of the 
authorizing statute? 



Meeting the Purposes of the Authorizing Statute is the first and broadest 
criterion you must consider. In most cases, it will not be given a separate 
evaluation on the Technical Review Form or assigned a point value, but you 
should keep it in mind as an overall consideration throughout your entire 
review (particularly when assessing Extent of Need for the Project). 

The program office you review for will acquaint you with the purposes of the 
statute before you begin your review. This criterion is meant to ensure that 
funding awards are in keeping with legislated educational priorities. 
Regardless of the quality of its response to the other selection criteria, an 
application that does not serve the purposes outlined in the authorizing 
statute should not receive funding. 

The following abbreviated version of an authorizing statute's purpose applies 
to the application passages that form the basis of the exercises provided in 
this module. Please read it carefully. There is no practice exercise for this 
criterion, but you should consider the information presented on the next page 
when completing the practice exercises for the other selection criteria. 
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Authorizing Statute : The Carl D. Perkins Vocational Education Act of 1984 

The statute allocates funding for Indian and Hawaiian Natives projects that 
target education, medical attention, relief of distress, and social welfare. A 
certain percentage of these funds must be used to provide vocational 
education services designed to meet the special needs and enhance the 
participation of handicapped individuals, disadvantaged individuals, adults in 
need of training or retraining, single parents or homemakers, and others. 
The 24 purposes identified by the statute include the following: 

1) The advancement of vocational education programs designed to 
improve the quality of vocational education, including high- 
technology programs involving an industry-education partnership; 

2) The expansion of vocational education activities necessary to 
meet student needs, and the introduction of new vocational 
education programs, particularly in economically depressed urban 
and rural areas; 

3) The creation or expansion of programs to train workers in skilled 
occupations needed to attract or revitalize businesses and 
industries; 

4) The improvement and expansion of postsecondary and adult 
vocational education programs and related services for out-of- 
school youth and adults; 

5) The acquisition of equipment, including high-technology 
equipment, and the renovation of facilities necessary to improve 
or expand vocational education programs; 

6) The creation or improvement of placement services for students 
who have successfully completed vocational education programs; 
and 

7) The provision of vocational education through arrangements with 
private vocational education institutions and employers, if these 
can make a significant contribution to project objectives or 
provide equivalent or superior training not available at a public 
vocational education institution. 
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Criterion: Extent of Need for the Project 



As noted in Meeting the Purposes of the Authorizing Statute, the statute that 
authorizes a particular program has certain declared purposes. These 
purposes derive from perceived educational needs. In order to qualify for 
funds, an applicant must specify the needs that the proposed project will 
meet, and those needs must be consistent with the purposes of the program 
providing the funding. 



When you review an application's response to this criterion, you must ask: 

1) What needs are outlined by the authorizing statute? 

2) What needs does the applicant identify? 

3) How did the applicant identify those needs, i.e., what 
specific documentation or evidence does the application offer 
to support the applicant's assessment of need? 

4) Are the needs identified by the applicant consistent with the 
purposes of the authorizing statute? 

5) Does the applicant identify too many or too few needs for the 
proposed time frame and resources of the project? 

6) Are the outlined needs well defined so that the project can be 
focused on them, or are the outlined needs very generic? 



The exercise on the next page is designed to give you practice in identifying the strengths 
and weaknesses of an application's response to the Extent of Need for a Project criterion. 
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EXERCISE: EXTENT OF NEED FOR THE PROJECT 

Directions: Read through the following passage* on Extent of Need for the Project. As you 
read, you should take note of significant strengths and weaknesses and list them in the 
space provided on 11-13. Use the overview at the end of the module to guide your review. 



Estimated Reading Time: 5 minutes 



The Emanon Tribe has 2,700 people and has lived in the Odega Foothills since the forced 
resettlement of 1869. Tribal members have lived in poverty for the better part of this 
century and, until recently, have had few prospects for change. Our need, however, is clear: 
to endl the long cycle of dependency resulting from low income levels and poor living 
conditions. 

A. CURRENT SITUATION 



1. Employment and Income 

The current unemployment rate for the Emanon Tribe is 69 percent, even though the 
unemployment rate for the greater Cimmaron Metropolitan Area (CMA), which ends only 
20 miles from the reservation, is only 6.3 percent. Not surprisingly, the median income for 
our 375 tribal families is $4,350, well below both Federal poverty levels and the median 
CMA income of $16,860. As a result of these factors, Emanon tribal members depend 
heavily on Federal and State economic aid. Even those who have gotten jobs have usually 
had to leave the reservation to find them, and then often that income is spent in the city, 
farther draining any economic base in the Foothills. 



The long-term future of the Emanons thus rests on the development of jobs and businesses 
on the reservation. 



2. Natural Resources and Development 

The Emanon Reservation includes 8,243 acres. Until very recently, this large land area was 
effectively isolated from the CMA, though only 20 miles away. However, the recent 
completion of Interstate-72 means a major change. The Interstate has two exits adjacent to 
the Reservation and runs the entire 6-mile length of the Emanons' land before cutting back 
northeast and upstate. 

In short, the Emanon Reservation and surrounding lands are now poised for major 
economic development. Construction of homes and shopping areas has already started 



* All the passages used for the selection criteria exercises in this module are from a single 
application. Therefore, you should keep in mind the information already received as you 
move from exercise to exercise. 
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north of the CMA and projects have already been initiated within a 10-mile radius of the 
Reservation. The Emanon Tribal Council has been approached by three developers in the 
past year, but has been reluctant to make commitments without a fully developed economic 
plan. 

That problem was recently solved with the help of a $125,000 Economic Development 
Administration (EDA) planning grant. That grant allowed us to develop a Comprehensive 
Development Plan (CDP) for the area. 

B. PLANS FOR THE FUTURE 

1. The CDP and Vocational Education 

The CDP, which was endorsed by the Tribal Council this spring, calls for a gradual but 
steady development of small businesses and light industry on the 22 percent of Emanon 
land closest to the Interstate. In addition, the Plan calls for the construction of housing and 
associated shopping areas west of the economic development. A second EDA grant for 
$1.4 million was recently awarded for the first phase of business development, the 
construction of a light industrial park to include printing and electronics assembly plants. 
These two businesses are growth areas in the greater CMA and are projected to remain so 
due to development along 1-72 (see Attachment A, Emanon Comprehensive Development 
Plan). In addition, the Tribal Council recently approved a contract with Whitman 
Construction for a planned community of 100 small, subsidized ranch homes over the next 
three years. This contract requires that at least 50 percent of construction jobs over that 
period (estimated at 100 workers) be given to Emanon tribal members, provided they meet 
minimum training requirements. 

A priority of the CDP is assisting tribal members to achieve their individual career goals in 
"demand" occupations on the reservation and in the surrounding nontribal community. 
With the help of Cimmaron City University, the Cimmaron Center for Business and 
Economic Research, and the Cimmaron Private Industry Council, we have focused our 
proposed education program on occupations that are undersupplied in the area. In 
choosing nontribal areas of vocational education, we have considered statements of 
longstanding need and intent to hire. We have targeted three employment areas for our 
initial development: printing, electronics assembly, and carpentry. Preparation of tribal 
members for those jobs, however, depends heavily on vocational education. Because job 
opportunities in these areas are projected to grow over the next 10 years (see the CDP), we 
believe that giving Emanons job skills in these areas will continue to provide stable 
employment. However, if our members cannot compete for these jobs, development will 
continue without the Emanons, leaving us in the poverty that has characterized our 
existence for four generations. 

Our proposed education project will allow us to provide a program uniquely designed to 
meet the special needs of the Indian community. A reservation-based program for Indians 
by Indians can serve a number of needs that cannot be filled by other educational 
institutions or programs that already exist in the Valley area: (1) individual attention, (2) 
personalized services, (3) a comfortable learning atmosphere, (4) community support, (5) 
easy access to classes, and (6) placement and other followup services. 
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2. Nontribal Employment 

In addition to the three occupations identified above, we propose to train workers for the 
growing tourism industry as well. The CMA Development Board has established growth of 
the Foothills Recreation Area, a 1,015-acre development that extends almost to the start of 
the Reservation, as a major goal now that 1-72 is completed. In conjunction with the 
Cimmaron Chamber of Commerce, the Board has adopted a plan for employment growth 
in the Recreation Area. The Emanons' knowledge of the Foothills' history and natural 
resources gives us an advantage for many of these jobs if we have the skills. 

Thus, we see jobs in tourism as a fourth occupational area to target through this project. 
Such off-reservation jobs will play a major part in building an integrated economic base 
with the CMA. However, the Emanon people have been unable to obtain or retain such 
jobs in the past. There has traditionally been a high "dropout" rate for nontribal 
employment. To some extent, this may be due to Emanons' inability to deal with the white 
"establishment" and the conflicting norms and practices that face Emanon workers in the 
predominant culture. These factors must also be addressed in the educational program in 
order to assure the longrun economic success of the tribe. If we are able to do this, along 
with the tribal employment gains we foresee, we may be able to break the cycle of poverty 
that has plagued our people and thus end our dependence on government financial 
assistance. 
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List the significant strengths and weaknesses of the preceding passage on Extent of Need 
for the Project in the space provided below. Tiy to indicate the relevant page number(s) 
for each. 



Strengths 



Weaknesses 



Compare your list to the list of selected strengths and weaknesses on the next page. 
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SELECTED STRENGTHS AND WEAKNESSES 



Strengths 

■ Applicant provides documented 
evidence of need to improve 
employment and income 
situation. (II- 10, Al) 

■ The goal of the proposed project 
is clearly specified. (11-10, Al j 



■ The applicant details how the 
economic plan will improve the 
tribal situation. (II- 10, A-l) 

■ The project hopes to make future 
government assistance 
unnecessary. (11-12) 

■ Proposed vocational education 
effort is tied into larger economic 
development effort, fll- 11, Bl) 



■ Education program plans are not 
focused narrowly on reservation 
jobs, but address the larger need 
to equip Indians to work in jobs 
outside the reservation. (II- 1 1 to 
11 - 12 ) 

■ Applicant's proposal responds to 
needs clearly documented by 
outside institutions. (II- 11, Bl) 

■ Finding off-reservation jobs for 
Emanons is clearly possible. 

(11-11 to 11-12, Bl-2) 

■ Reliance on statements of 
longstanding need and intent to 
hire indicates realistic objectives. 
(11-11, Bl) 

■ Applicant identifies needs that 
are unique to the target group, 
although these needs are not very 
specific or well documented. 
(11-11, Bl) 



Weaknesses 

Assessment does not specify the kind and 
number of jobs to be provided by the 
project. (11-11 to 11-12, Bl-2) 



It is unclear what skills the target group 
lacks that make them unable to find work. 
(11-11, Bl) 

The application lacks specific data 
indicating the availability of jobs. (II- 11 to 
11 - 12 , Blh) 

Little detail is provided on how the 
project will move trainees into the work 
environment. (II- 11 to 11-12) 

Applicant needs to provide statistics to 
support claim of high dropout rate for 
those employed outside the reservation. 
( 11 - 12 ) 
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Criterion: Plan of Operation 



Having outlined the need for a project, the applicant must detail the project's 
objectives. The applicant must also provide evidence that the project will be 
efficiently and effectively operated to achieve those objectives. 



When you review an application's response to this criterion, you must ask: 

1) Do the project objectives serve the purposes of the authorizing 
statute? 

2) How well is the project designed? 

Are project objectives consistent with stated needs? 

Are project activities consistent with project objectives? 

Are project objectives measurable? 

3) How will the applicant use its resources and personnel to achieve 
each objective? 

4) Has the applicant developed an effective management plan that 
will ensure proper and efficient administration of the project? 

5) Do project milestones represent a logical progression of times 
and tasks? 

6) Does the applicant propose a realistic time schedule for 
accomplishing objectives? 

7) Will the proposed activities accomplish the project's 
objectives successfully? 

8) Are the educational approaches planned based on sound 
research that indicates they will oe successful for the 
population to be served? 

9) Does the project have clearly developed provisions for providing 
equal access to eligible participants who are members of 
traditionally underrepresented groups (racial or ethnic minorities, 
women, handicapped persons, elderly persons)? 



Now work through the exercise on the next page. This next exercise is designed to give you 
practice in identifying the strengths and weaknesses of an application's response to the 
Plan of Operation criterion. 
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EXERCISE: PLAN OF OPERATION 

Directions: Read through the following passage on Plan of Operation. As you read, you 
should take note of significant strengths and weaknesses and list them in the space 
provided in 11-20. Use the overview at the end of the module to guide your review. 



Estimated Reading Time: 9 minutes 



A. PROJECT GOALS 

Our overriding goal is to increase the proportion of economically independent Emanon 
families. We propose to do this in two ways: tribal and off-reservation employment 
resulting from a strong program of vocational education. 

More specifically, the project has four goals: 

1. We will decrease the tribal unemployment rate from 69 percent to 12 
percent over the three-year project (see Attachment B for documentation 
on tribal unemployment figures). 

2. We will place 85 percent of our graduates into jobs in their field of 
training within 3 months of graduation from the program. 

3. We will increase the number of Emanons in off-reservation employment. 

4. We will increase Emanon self-esteem as a result of job placement and 
economic development. 

B. OBJECTIVES 

The project has 5 objectives: 

1. To train 20 workers per year for entry-level jobs in commercial 
printing (see Attachment C(l) for job projections); 

2. To train 20 workers per year for entry-level jobs as electronic 
parts assemblers (see Attachment C(2) for job projections); 

3. To train 20 workers per year for entry-level carpentry positions 
(see Attachment C(3) for job projections and statement by the 
Cimmaron Federation of Building and Trades Unions); 

4. To train 15 workers per year for jobs as tour/travel managers (see 
Attachment C(4) for job projections); and 

5. To provide job readiness training. 
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C. CURRICULUM 

1. Objective #1 : To train 20 workers per year for commercial 

printing jobs. 

Cimmaron Technical College, located 23 miles south on 1-72, has 
a one-year, 30-credit vocational training program that can be 
adapted to our needs. Discussions with the College indicate that 
they are willing to modify the program to suit our students. The 
course of study includes an overview of the commercial printing 
field and instruction in the four major commercial printing 
systems commonly used throughout the CMA (the Wainwright X- 
365, X-470, Z-7200, and the Natuki Master 5000). An internship 
in a CMA printing facility is included toward the end of the 
curriculum. Graduates of this program will qualify for the 
Printing Technician-Level 1 positions anticipated in the Emanon 
Industrial Park facility under our Comprehensive Development 
Plan. 

2. Objective #2 : To train 20 workers per year for electronic parts 
assembly jobs. 

Foster-Johnson, the largest electronic parts firm in the CMA, has 
an in-house training program for Parts Assembler positions. The 
program includes a total of 320 hours of instruction, and Foster- 
Johnson has agreed to provide a resident instructor in about 4 
months. Classes will be held in the Emanon Tribal Hall until the 
company's new plant opens next year. The project must handle 
the financial arrangements for the instructor and classroom 
materials, although Foster-Johnson has agreed to provide all 
equipment. Graduates of the class will be given entry-level 
positions when the plant opens (in about 9 months). 

3. Objective #3 : To train 20 workers per year for carpentry jobs. 

A curriculum for Carpentry Assistant positions has been 
developed by the Cimmaron Federation of Building and Trades 
Unions, but staff and facilities to conduct it must be secured. We 
propose to hire an instructor on a part-time basis for this purpose, 
ana we will construct a small classroom facility near the Tribal 
Hall. We anticipate that the 400-hour training program can begin 
within 6 months, producing its first graduates within 11 months. 
(Construction by the Whitman company is due to begin in 10 
months, after site-clearing has been finished.) We anticipate that 
the curriculum will need some modifications based on our 
students' current skills, but we are prepared to make them. 
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4. Objective #4 : To train 15 workers per year for jobs as tour/travel 
managers. 

The curriculum for this course will be developed under the 
guidance of an advisory group consisting of representatives from 
the Cimmaron Chamber of Commerce, the Foothills Community 
College Business Development Program, and the Cimmaron 
Development Board. Areas of training are expected to include 
travel management, tour development, small business 
management, and marketing. Graduates of the program will be 
eligible for employment under an arrangement between the 
Cimmaron Development Board, the Tribal Council, and Resorts 
International, a national tour center chain that recently 
established a northwest hub in Cimmaron. 

5. Objective #5 : To provide job readiness training. 

The job readiness program will include an 80-hour course of study 
for all Emanon project participants. The curriculum will, when 
completed, cover such areas as human development, 
employability skills, and job survival skills. The curriculum will be 
developed and delivered by a counselor employed by the project. 

Completion of this course will be required before placement in 
one of the job skill programs described above. 

D. STUDENT RECRUITMENT 

Students will be recruited throughout the Emanon Reservation using a variety of means 
including personal contact, the tribal newsletter, and announcements by WKFH, the local 
radio station. We will also develop a flyer on the project and post it at local businesses and 
in the Tribal Hall. 

E. PROJECT MANAGEMENT 

Management of the project will be handled by a Project Director, who will be selected by 
and accountable to the Tribal Council. The Project Director will be in charge of all aspects 
of the program, including program development, supervision of all staff and instructors, job 
placement, student recruitment, and fiscal matters. The Director will also be responsible 
for completing all reports to the U.S. Department of Education and for liaison with local 
agencies, associations, and employers. 

In all cases, the provisions of the Education Department General Administrative 
Regulations, CFR 34, will be adhered to. Equipment purchased and materials acquired 
during the project will be solely reserved for project purposes, and inventories will be 
maintained by the Project Director. A certified public accountant retained by the tribe will 
audit all records of the project and assure compliance with all Federal audit standards. 

A Project Secretary will be hired to provide administrative support for all project activities. 
In addition, a part-time counselor will be hired to handle the job readiness component and 
to provide career guidance. Part-time instructors will be hired as described above. 
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F. TIME FRAME 

The five training aspects of the program have been scheduled to coincide with employment 
opportunities emerging in the area: 

1. The job readiness training program will begin in July to assure 
that participants complete it in time to enter skill trai nin g 
programs. 

2. The commercial printing classes will begin in September with the 
first semester at Cimmaron Technical College. This should allow 
the first graduates to be ready for the opening of the Emanon 
Industrial Park facility. 

3. The electronic parts assembly training program will begin in the 
fall as well, also to assure that graduates emerge in time for the 
opening of the Industrial Park. 

4. The carpentry program will begin in February and produce its 
first graduates next July, shortly after construction begins on the 
Whitman tract. 

5. The tour/travel program will also begin in February. The training 
schedule for this area should not be a problem, however, as major 
growth of the Recreation Area is not expected to begin for at 
least another year. 

These dates will be adhered to and made possible by a number of startup activities 
undertaken early in the project period. The Emanon Tribal Council is strongly committed 
to this project and will ensure that training programs are not delayed. 
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List the significant strengths and weaknesses of the preceding passage on Plan of 
Operation in the space provided below. Tiy to indicate the relevant page number(s) for 
each. 



Strengths 



Weaknesses 



Compare your list to the list of selected strengths and weaknesses on 11-22. 
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SELECTED STRENGTHS AND WEAKNESSES 



Strengths 



Weaknesses 



Evidence demonstrates that the 
CDP has been found viable by 
other grant-making organizations 
(See Need, II- 11, A2) 



■ A close relationship between 
training, availability of jobs in the 
area, and employer needs is 
yrogosed. (11-16, Bl-4; 11-17, C2; 

■ The first three goals are 
measurable. (11-16, Al-3) 

■ Exact number of persons to be 
trained is given. (11-16, Bl-4; II- 
17 to II-lC Cl-4) 

■ The first- four objectives are 
detailed and documented. (11-16, 
Bl-4) 



■ Training objectives relate to the 
overall economic plans of the 
tribe. (11-16, Bl-4; 11-17, Cl) 



■ Resources are detailed when 
necessary. (11-17 to 11-18, Cl-4) 

■ Responsibilities of key personnel 
are outlined. (11-18, E) 



Not clear how "dropout" rate for off- 
reservation activities is to be cured by 
vocational training. How will "dropout" 
attitude be attacked specifically? 

(See Need, 11-12) 

First goal seems very optimistic - can 
proposed project effect such a drastic 
reduction in unemployment given the 
diverse factors that lead to unemployment 
in this population? (11-16, Al) 

Goal 4 is not measurable as stated. (11-16, 
A4) 

Specific objectives for the job readiness 
component are not clear. (11-16, B5) 



There seems to be no plan to assess the 
basic skills of the students and to tailor 
instruction to their skill levels. (11-17 to II- 
18, C-D) 

The educational methodology to be used 
with adults who have low skm levels and 
low self-esteem is not clear. (11-17 to II- 
18, C) 

Recruitment procedures are not specific. 
(11-18, D) 

Selection procedures are not described. 
(11-18, D) 

Burden of Project Director is too heavy. 
(11-18, E) 

Claim that accounting and audit system 
will be satisfactory by Federal standards 
should be documented. The nature and 
extent of oversight should be clarified. (II- 
18, E) 

The time line is very general. Although 
some details about the number of 
classroom hours, etc., are given in the 

E lan of operation narrative, it is not clear 
ow all activities will fit together and be 
managed effectively. (11-19, F) 

Application fails to detail startup 
activities. (11-18, D; 11-19, F) 
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Criterion: Quality of Key Personnel 



Applicants must outline the staff required for the project, identify personnel 
to fill these staff roles, and detail the relevant qualifications of these 
personnel. 



When you review an application's response to this criterion, you must ask: 

1) Do the job descriptions adequately reflect skills needed to 
make the project work? 

2) Are the duties of personnel clearly defined? 

3) What relevant qualifications do the proposed personnel possess, 
especially the Project Director? (Focus on their experience 
and training in fields related to the objectives of the project, 
though other information may be considered.) 

4) Will proposed personnel need to be trained for the project? 

5) How much time will the proposed personnel actually devote to the 
project? 

6) To what extent does the applicant encourage employment 
applications from members of traditionally underrepresented 
groups (ethnic or racial minorities, women, handicapped persons, 
elderly persons)? 



Now work through the exercise on the next page. This next exercise is designed to give you 
practice in identifying the strengths and weaknesses of an application's response to the 
Quality of Key Personnel criterion. 
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MODULE II: Selection Criteria 



EXERCISE: QUALITY OF KEY PERSONNEL 

Directions: Read through the following passage on Quality of Key Personnel. As you read, 
you should take note of significant strengths or weaknesses and list them in the space 
provided on 11-25. Use the overview at the end of the module to guide your review. 



Estimated Reading Time: 3 minutes 



A. Project Director: Annemarie C. Max 

Tribal Education Director Annemarie C. Max has served as director of two previous 
Federal grants (Grant #G000000406 for the Tribe's adult vocational education program 
and Grant #G0000010312 for the Tribe's adult literacy project). Both projects were 
funded by the U.S. Department of Education and fully met their original objectives. 
Reports for both are on file and are summarized in Attachment D. Ms. Max has a B.A. 
degree in education from the University of Idaho and over 9 years' work experience in 
tribal and other educational programs. She is a seasoned manager. (See Resumes, 
Attachment E) 

B. Project Secretary: Marlena G. Amberson 

Ms. Amberson is a successful graduate of our 1983 vocational education class and has 
served with the Tribal Council staff ever since. She has performed well and is a dedicated 
member of the Tribe. (See Resumes, Attachment E) 

C. Vocational Counselor: Carla Redbird 

Ms. Redbird has a Master of Science degree in educational and vocational counseling, with 
an emphasis on career guidance. She is a graduate of Cimmaron University and has served 
as a school counselor for 6 years. She is sensitive to the needs of the Emanons who are the 
target group for this project, as evidenced by her role as the counselor in our previous adult 
literacy project. (See Resumes, Attachment E) 



MODULE II: Selection Criteria 



List the significant strengths and weaknesses of the preceding passage on Quality of Key 
Personnel in the space provided below. Try to indicate the relevant page number(s) for 
each. 



Strengths 



Weaknesses 



Compare your list to the list of selected strengths and weaknesses on the next page. 
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SELECTED STRENGTHS AND WEAKNESSES 



Strengths 

■ Project Director has prior, 

relevant experience supervising a 
vocational education program. 
(H-24, A) 



■ Secretary is a graduate of one of 
Tribe's previous vocational 
education programs. (11-24, B) 

■ Counselor has previous 
experience in educational 
counseling. (11-24, C) 



Weaknesses 

Amount of time personnel will devote to 
the project is not given. (11-24) 

Job descriptions are not provided. Hard 
to judge if personnel have the skills 
necessary to their proposed roles. (11-24) 

Other personnel, such as instructors* are 
not listed. Applicant does not explain 
how these personnel will be selected or 
what methods will be used to ensure that 
chosen instructors are sensitive to the 
special needs of the Indian population 
they will teach. (11-24) 



■ Resumes are provided. 
(Attachment E) 



MODULE II: Selection Criteria 



Criterion: Budget and Cost Effectiveness 



Applicants must provide a detailed budget that is adequate to carry out the 

S roject as well as cost effective. The budget should indicate how Federal, 
tate, and local funds will be used, in addition to any other funding, and 
include the proposed salaries of project personnel. 



When you review an application's response to this criterion, you must ask: 

1) Is the budget adequate to support the project's proposed 
activities? 

2) Are overall project costs reasonable in relation to project 
objectives? 

3) How much of the project's total cost is devoted to 
administrative costs? 

4) Are budget items sufficiently justified? 

5) Is the budget padded? 



Now work through the exercise on the next page. This next exercise is designed to give you 
practice in identifying the strengths and weaknesses of an application's response to the the 
Budget and Cost Effectiveness criterion. 



MODULE II: Selection Criteria 



EXERCISE: BUDGET AND COST EFFECTIVENESS 

Directions: Read through the following passage on Budget and Cost Effectiveness. As you 
read, you should take note of significant strengths or weaknesses and list them in the space 
provided on 11-31. Use the overview at the end of the module to guide your review. 



Estimated Reading Time: 6 minutes 



The budget for each year of the 3-year project, including special one-time startup costs, is 
shown below: 



FIRST-YEAR 



PERSONNEL 






1 . 


Salaries (Full-time staff) 




$43,000 




Project Director (12 months) 
Project Secretary (12 months) 


$33,000 

$10,000 




2. 


Fringe Benefits ((5)20%') 




$ 8,600 




Project Director 
Project Secretary 


$ 6,600 
$ 2,000 




3. 


Part-time Wages 




$42,263 




Counselor: 30 hours for 26 weeks 
x $15/hr = $11,700 plus 10 hours 
for 26 weeks = $3,900 plus 
fringe @ 15% = $2,340 


$17,940 






Instructor-Electronics: 320 hours 
x $20/hr = $6,400 plus fringe 
@ 15% = $960 


$ 7,360 






Instructor-Carpentry: 500 hours 
x $ 19/hr = $9,500 plus fringe 
@15% =$1,425 


$10,925 






Instructor-Tour /Travel: 350 hours 
x $ 15/hr = $5,250 plus fringe 
@15% = $788 


$ 6,038 
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B. TRAVEL $12,800 

1. To U.S. Department of Education $ 2,800 

annual project directors' meeting in 
Washington, D.C.-for Project 

Director and counselor 

2. Printing Program-travel of students $10,000 

to Cimmaron Technical College-40 

weeks x $250 per week for bus/driver 

C. EQUIPMENT (STARTUP) $12,065 

Wilding Band-Saw $ 1,250 

Operman Jig-Saw $ 895 

Pownell Hack-Saw $ 300 

Emerson Hand-Saws (10 @ $12) $ 120 

Hand Tools (hammers, screwdrivers, $ 700 

@ $35 per student) 

Operman C-Lathe $ 2,850 

Operman Soldering Irons (20 @ $15) $ 300 

Classroom and Office Furniture 

- File Cabinets (4 @ $200) $ 800 

- Student Desks (20 @ 100) $ 2,000 

- Teacher Desk (1 @ $150) $ 150 

- Utility/Work Tables (5 @ $250) $ 1,250 

- VCR for job readiness program, $ 1,450 

including monitor and camera 

D. SUPPLIES $12,750 

Electronic Parts Program: 

Textbooks, demonstrators, solder, $ 3,500 

part assemblies, etc. 

Carpentry Program: 

Wood, nails, screws, paint, finish, etc. $ 2,750 

Tools @ $200/Student $ 4,000 

Job Readiness Program: 

Counseling materials, instruments, etc. $ 1,500 

Office Supplies $ 1,000 
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E. CONTRACTS $59,000 

Tuition to Cimmaron Technical College 
($2,700 tuition plus $50 registration 
plus $200 materials fee per student x 20) 

F. STUDENT STIPENDS $138,690 

Commercial Printing: 40 weeks 

x 30 hours x $3.35 x 20 students $80,400 

Electronics: 10 weeks x 32 hrs 

x $3.35 x 20 students $21,440 

Carpentry: 10 weeks x 40 hours 

x $3.35 x 20 students $26,800 

Tour/Travel: 200 hours x $3.35 $10,050 

x 15 students 



G. OTHER 



$11,325 



Job Readiness Consultant: 4 sessions 



of 10 workshops $ 5,000 

Local Travel: 1,000 miles @ $ .225 $ 225 

Student Insurance $ 2,500 

Duplicating $ 1,500 

Telephone $ 600 

Postage/Printing $ 1,200 

Office Supplies $ 300 



H. TOTAL DIRECT COSTS 

I. INDIRECT COSTS (@ 8%) 



$340,493 
$ 27.239 



TOTAL PROJECT COST (FIRST YEAR): 



$367,732* 



* Note to Reviewers: In the interest of brevity, second- and third-year costs have been 
omitted. 
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MODULE II: Selection Criteria 



List the significant strengths and weaknesses of the preceding passage on Budget and Cost 
Effectiveness in the space provided below. Try to indicate the relevant page number(s) for 
each. 



Strengths 



Weaknesses 



Compare your list to the list of selected strengths and weaknesses on the next page. 



O 

ERIC 
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MODULE II: Selection Criteria 



SELECTED STRENGTHS AND WEAKNESSES 



Strengths 

■ Applicant provides 
comprehensive cost assessment 
for all project years. (11-28 to II- 
30) 

■ Salaries for Project Director and 
Secretary seem reasonable. (II- 
28, Al) 



Weaknesses 



Budget offers no narrative justification. 
For example, are instructor hourly rates 
at prevailing area wages? (11-28, A3) 

Fringe benefits are not itemized or 
justified. (11-28, A2) 

Is the counselor's presence at the 
conference necessary? (11-29, Bl) 

Do carpentry students require their own 
set of tools? Couldn't these be shared by 
all the classes? (11-29, C) 

Cost of $5,000 for job readiness 
consultant seems nigh. (11-30, G) 

No effort seems to have been made to 
obtain contributions from the Tribe or 
businesses. (11-28 to 11-30) 

Project intends to train 75 students, which 
makes the project cost per student around 
$4,900 per year. Amount seems high. (II- 
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Criterion: Evaluation Plan 



Applicants must propose a plan to evaluate how well project activities fu lfill 
project objectives. The proposed methods of evaluation should be detailed. 
Applicants should explain how interim evaluation data will be used to 
improve project performance and how final evaluation of the project will 
determine how well project purposes were achieved. 



When you review an application's response to this criterion, you must ask: 

1) Are the proposed methods of evaluation appropriate to the 
project? 

2) Will the proposed evaluation be objective? 

3) Will the proposed evaluation methods measure the 
effectiveness of project activities in meeting project 
objectives? 

4) Will the evaluation plan produce valid and reliable data 
concerning the accomplishment of project objectives? 

5) Does the evaluation plan measure the project's effect on 
the project audience? 



Now work through the exercise on the next page. This next exercise is designed to give you 
practice in identifying the strengths and weaknesses of an application's response to the the 
Evaluation Plan criterion. 
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EXERCISE: EVALUATION PLAN 

Directions: Read through the following passage on Evaluation Plan. As you read, you 
should take note of significant strengths or weaknesses and list them in the space provided 
on 11-35. Use the overview at the end of the module to guide your review. 



Estimated Reading Time: 3 minutes 



A. STUDENT LEARNING 

The instructor for each vocational program area will administer tests to determine the level 
of student learning. These tests will be shared with the counselor, who will meet regularly 
with each student to review progress and provide help. A special instrument will be used to 
assess the effectiveness of the human development component of the job readiness 
training. 

In addition, the Project Director will monitor weekly class attendance. Needed counseling 
will be provided. 

B. PROCESS DATA FROM INSTRUCTORS 

Faculty meetings will be held on a regular basis to review progress and note any special 
problems that must be addressed. The Project Director and counselor will be present at 
each meeting. In addition, spot visits to classes will be conducted by the Project Director. 
(These will be unannounced.) 

C. STUDENT COMMENTS 

Students will be surveyed using a brief survey form developed and used by the Project 
Director with -good success in an earlier grant project. The form will obtain comments on 
the program and suggestions for improvement. These will be reviewed by the Project 
Director and counselor, and needed project changes will be made. 

D. PLACEMENT 

Placement statistics will be maintained for each class as they finish. In addition, a followup 
will be conducted with each program graduate to determine employment status and success 
6 months after program completion. In addition, we will obtain feedback from employers 
on project success. 

These evaluation activities are the responsibility of the Project Director and will be 
implemented by her with assistance from the counselor. 
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MODULE II: Selection Criteria 



List the significant strengths and weaknesses of the preceding passage on Evaluation Plan 
in the space provided below. Tiy to indicate the relevant page number(s) for each. 



Strengths 



Weaknesses 



Compare your list to the list of selected strengths and weaknesses on the next page. 
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MODULE II: Selection Criteria 



SELECTED STRENGTHS AND WEAKNESSES 



Strengths 

■ Evaluation procedures include 
instructor feedback. (11-34, A) 

■ Evaluation procedures include 
student feedback. (11-34, C) 

■ Student survey form was used 
with prior success. (11-34, C) 

■ Evaluation plan includes 
feedback from employers as to 
effectiveness of activities. (11-34, 
D) 

■ Followup evaluation will measure 
successful accomplishment of 
objectives. (11-34, D) 

■ An evaluation instrument is 
included to measure the effect of 
the human development 
component of the job readiness 
workshops. (11-34, A) 



Weaknesses 

Evaluation instruments are not detailed. 
Who will prepare them? Need more 
information on the instruments and 
methods that will be used to measure 
student progress in the vocational 
education areas of the project. (11-34, A, 
C, D) 

No indication of how students will be 
monitored or how counseling will be 
coordinated. (11-34, A-B) 

The method and frequency of instructor 
feedback is not stated. (11-34, A-B) 

Applicant should include a copy of the 
survey form for review. (II-34, C) 

It is not clear how survey data will be 
used to improve instruction. (11-34, C) 

The method and frequency of feedback 
from employers is not clear. (11-34, D) 

Evaluation plan does not explain how 
placement data will be reported. (11-34, 

A majority of the evaluation is ongoing 
and relies heavily on the subjective 
assessment of project staff. (11-34, A-C) 

Plan does not indicate how program 
improvement needs will be identified. (II- 
34f D) 

No provision is made for outside 
evaluation to assure the objectivity of 
internal evaluation. (H-34) 

The reliability and validity of evaluation 
data is not discussed. (11-34) 

The budget includes no provision for 
purchasing or scoring evaluation 
instruments. (II-29, D; 11-30, G) 
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MODULE II: Selection Criteria 



Criterion: Adequacy of Resources 



Applicants must identify the facilities, equipment, and supplies they propose 
to use during the project and show that these are adequate to achieve project 
objectives. Resources also include involved personnel or any organizations 
or groups that will contribute to the project. 



When you review an application’s response to this criterion, you must ask: 

1) Are the proposed facilities adequate for project purposes? 

2) Is the proposed equipment adequate for project purposes? 

3) Does the applicant have access to special sources of 
experience or expertise? 



Now work through the exercise on the next page. This next exercise is designed to give you 
practice in identifying the strengths and weaknesses of an application's response to the the 
Adequacy of Resources criterion. 
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MODULE II: Selection Criteria 



EXERCISE: ADEQUACY OF RESOURCES 

Directions: Read through the following passage on Adequacy of Resources. As you read, 
you should take note of significant strengths or weaknesses and list them in the space 
provided on 11-39. Use the overview at the end of the module to guide your 
review. 



Estimated Reading Time: 3 minutes 



A. EXPERIENCE OF KEY STAFF 

The Emanon Tribal Council has demonstrated its capability for successful management of 
Federal education grants on two previous occasions. In addition, the proposed staff have 
demonstrated their expertise through previous project management and support activities. 
Still further, the Tribal Council will exercise direct oversight on this project through 
quarterly reports submitted by the Project Director and the use of its certified public 
accountant. 

B. FACILITIES 

The Emanon Tribal Hall has adequate space to house the onsite classrooms, except for the 
carpentry class. As indicated, this facility will be constructed-at no cost to the project. 
Project staff will be housed in the Tribal Hall offices and thus have easy access to students 
and instructors. The Tribal Hall was modified through a previous grant to permit access of 
handicapped individuals and is within easy commuting distance of prospective students. 

All Tribal Hall facilities can be secured and are protected by an alarm system. In addition, 
an inventory of all project equipment will be maintained. 

C. OTHER 

As indicated above, some of the instruction will occur at the Cimmaron Technical College, 
an accredited 2-year institution built in 1985. The college meets all Federal standards for 
access and has adequate facilities and equipment for the printing curriculum. 

The project will be especially advantaged by close cooperation of employers in 
providing/designing instruction. Foster-Johnson Electronics, Inc., the employer of our 
electronics assembly graduates, will provide the instructor for the parts assembly program. 
The instructor for the carpentry program will be a certified union member to assure 
adequacy of instruction and high job prospects for our graduates. Cimmaron Technical 
College maintains close contact with area printing firms and thus should have no difficulty 
arranging for the needed internships. Their curriculum is carefully reviewed on a regular 
basis by an advisory committee composed of local printing firms. 
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MODULE II: Selection Criteria 



List the significant strengths and weaknesses of the preceding passage on Adequacy of 
Resources in the space provided below. Try to indicate the relevant page number(s) for 
each. 



Streng ths 



Weaknesses 



Compare your list to the list of selected strengths and weaknesses on the next page. 
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SELECTED STRENGTHS AND WEAKNESSES 
Strengths Weaknesses 



■ Facilities are within easy ■ 

commuting distance ana 

equipped tor handicapped 
persons. (11-38, B) 

■ 

■ Equipment and facilities are 
protected and secure. (11-38, B) 

■ Employer that will hire 
electronics assembly trainees will 
provide the instructor. (11-38, C) 

■ Available equipment seems 
adequate for training printers. 

(11-38, C) 



No detail provided on the physical 
suitability of facilities for work or training. 
(11-38, B-C) 

Facility for carpentry classes does not yet 
exist. (11-38, BJ 
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Conclusion 



All programs use these general selection criteria to a greater or lesser 
degree. Sometimes the criteria will be worded differently. Sometimes a 
criterion will not appear as a separate unit, but will be subsumed under 
another. And sometimes, additional selection criteria or narrower 
interpretations of criteria will be added by a program office through 
regulation. To avoid any possible confusion or oversight, be sure that you 
review any program-specific information provided by the program office you 
will read for. 



RULES OF THUMB FOR REVIEW 

1) What NEEDS justify the project? 

2) Will the project's OBJECTIVES meet these 
needs? 

3) Will the project's ACTIVITIES achieve these 
objectives? 

4) Will EVALUATION measure whether the 
objectives are in fact accomplished by the 
activities? 
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SELECTION CRITERIA OVERVIEW 



Meeting the Purposes of the Authorizing Statute 

1) What are the purposes of the authorizing statute? 

2) What are the objectives of this project? 

3) How will these objectives further the purposes of the authorizing statute? 

Extent of Need for the Project 

1) What needs are outlined by the authorizing statute? 

2) What needs does the applicant identify? 

3) How did the applicant identify those needs, i.e., what specific documentation or evidence does the 
application offer to support the applicant’s assessment of need? 

4) Are the needs identified by the applicant consistent with the purposes of the authorizing statute? 

5) Does the applicant identify too many or too few needs for the proposed time frame and resources of the 
project? 

6) Are the outlined needs well defined so that the project can be focused on them, or are the outlined needs 
very generic? 

Plan of Operation 

1) Do the project objectives serve the purposes of the authorizing statute? 

2) How well is the project designed? 

Are project objectives consistent with stated needs? 

Are project activities consistent with project objectives? 

Are project objectives measurable? 

3) How will the applicant use its resources and personnel to achieve each objective? 

4) Has the applicant developed an effective management plan that will ensure proper and efficient 
administration of the project? 

5) Do project milestones represent a logical progression of times and tasks? 

6) Does the applicant propose a realistic time schedule for accomplishing objectives? 

7) Will the proposed activities accomplish the project's objectives successfully? 

8) Are the educational approaches planned based on sound research that indicates they will be successful for 
the population to be served? 

9) Does the project have clearly developed provisions for providing equal access to eligible participants who 
are members of traditionally underrepresented groups (racial or ethnic minorities, women, handicapped 
persons, elderly persons)? 
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Quality of Key Personnel 

1) Do the job descriptions adequately reflect skills needed to make the project work? 

2) Are the duties of personnel clearly defined? 

3) What relevant qualifications do the proposed personnel possess, especially the Project Director? (Focus 
on their experience and training in fields related to the objectives of the project, though other information 
may be considered.) 

4) Will proposed personnel need to be trained for the project? 

5) How much time will the proposed personnel actually devote to the project? 

6) To what extent does the applicant encourage employment applications from members of traditionally 
underrepresented groups (ethnic or racial minorities, women, handicapped persons, elderly persons)? 

Budget and Cost Effectiveness 

1) Is the budget adequate to support the project's proposed activities? 

2) Are overall project costs reasonable in relation to project objectives? 

3) How much of the project's total cost is devoted to administrative costs? 

4) Are budget items sufficiently justified? 

5) Is the budget padded? 

Evaluation Plan 

1) Are the proposed methods of evaluation appropriate to the project? 

2) Will the proposed evaluation be objective? 

3) Will the proposed evaluation methods measure the effectiveness of project activities in meeting project 
objectives? 

4) Will the evaluation plan produce valid and reliable data concerning the accomplishment of project 
objectives? 

5) Does the evaluation plan measure the project's effect on the project audience? 

Adequacy of Resources 

1) Are the proposed facilities adequate for project purposes? 

2) Is the proposed equipment adequate for project purposes? 

3) Does the applicant have access to special sources of experience or expertise? 
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MODULE III: 



Writing Comments, Scoring, and Team Skills 
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Introduction 



This module is divided into four sections. The first section, The Technical Review Form, is 
an introduction to the Technical Review Form that you will use when you review 
applications. The next two sections discuss how to write comments and score an 
application. These sections also provide exercises to sharpen your skills. The last section, 
Team Skills, examines your responsibilities as a panel reviewer. This section provides tips 
on active listening and how to become an effective member of the panel team. Practice 
exercises designed to improve your team skills are included. 

This module builds on the analysis skills you learned in Module II, showing you how to turn 
identified strengths and weaknesses into thoughtful comments and appropriate scores. The 
lists of selected strengths and weaknesses identified in the sections on selection criteria 
form the basis of the writing and scoring exercises. Complete these exercises carefully so 
that you will be prepared to translate your analysis of applications into comments and 
scores on the Technical Review Form. Merely listing strengths and weaknesses will not 
constitute acceptable application review. 

As you may be asked to participate in a panel review, you should also read through the 
section on team skills. The listening and interpersonal skills that are discussed in this 
section can be applied to other situations as well, such as orientation sessions or meetings 
with program staff. 
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MODULE III: The Technical Review Form 



A. The Technical Review Form 



As you evaluate an application, you will provide written comments and 
numerical scores for each of the selection criteria. These comments and 
scores will be entered on a Technical Review Form that will be provided by 
the program office. Technical Review Forms will vary slightly from program 
office to program office, but all are comprised of a summary page and a 
comment/score sheet for each selection criterion. 

The summary page identifies the application under review and the reviewer. 
It also provides space to list numerical scores for each criterion, compute the 
final summary score, and write overall comments on the application's 
strengths and weaknesses. Each comment/score sheet identifies one of the 
selection criteria and lists relevant considerations for that criterion. Space is 
then provided for reviewers to comment on the strengths and weaknesses of 
an application's response to that criterion and to assign a numerical score 
chosen in consideration of those strengths and weaknesses. 

The criteria on the Technical Review Form may be the general EDGAR 
selection criteria outlined in Module II or modified criteria that reflect the 
particular emphases of the program office. As a reviewer, you should make 
sure you understand the selection criteria on the Technical Review Form 
before you begin your review. 
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MODULE III: Writing Comments 



B. Writing Comments 



Your comments should be based on the strengths and weaknesses of an 
application's response to the selection criteria. Although you may note a 
great many strengths and weaknesses as you read, you are not expected to 
comment on each one. In analyzing an application, certain strengths and 
weaknesses will seem more significant than others, and your comments 
should reflect these. You must use your professional judgment in 
commenting on major strengths and weaknesses, without, however, omitting 
any that are significant. 

When writing comments, be aware that they will be read by program staff as 
thorough evaluations of an application's response to the selection criteria. 
Comments should not merely describe what the application says; they must 
evaluate what is said in a concise and well-digested form. You are not being 
asked to turn thumbs up or down on the application, but to provide a specific 
and well-documented summary of the application's strong and weak points. 
Your comments must be tactful and constructive rather than frivolous or 
pejorative. Similarly, comments should provide specific information about 
which elements of the criteria are being considered instead of reiterating the 
wording of the selection criteria. When you point to a weakness or strength, 
you must indicate where that weakness or strength is located in the text by 
citing page numbers so that program staff do not have to search through the 
entire application. 

Be sure to differentiate comments based on fact from those based on 
professional judgment and to clearly distinguish comments based on lack of 
information from those based on information included in the application. 
Finally, write in legible and complete sentences, using proper grammar and 
spelling. 



GUIDELINES FOR WRITING COMMENTS 

1) Be specific. 

2) Evaluate rather than merely describe. 

3) Document your evaluation. 

4) Be tactful. 

5) Write legibly. 

6) Use complete sentences. 

7) Use proper grammar and spelling. 
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MODULE III: Writing Comments 



PREVIEW EXERCISE 

Directions: This exercise provides examples of good and poor comments for the Extent of 
Need for the Project criterion. Review guidelines for this criterion are given below, 
followed by a list of selected strengths that form the basis of the comments that follow on 
the next page. These strengths are drawn from the application passage provided on 11-10 
to 11-12. You may want to reread this passage to refresh your memory. 'Hie list of selected 
strengths is followed by a poor comment that is critiqued to identify its flaws. A good 
comment follows the critique. Read these comments and study the critique to prepare 
yourself for writing comments. 



Criterion: Extent of Need for the Project 



Review Guidelines 

1) What needs are outlined by the authorizing statute? 

2) What needs does the applicant identify? 

3) How did the applicant identify those needs, i.e., what specific 
documentation or evidence does the application offer to support the 
applicant's assessment of need? 

4) Are the needs identified by the applicant consistent with the purposes 
of the authorizing statute? 

5) Does the applicant identify too many or too few needs for the proposed 
time frame and resources of the project? 

6) Are the outlined needs well defined so that the project can be focused 
on them, or are the outlined needs very generic? 



Selected Strengths 



Applicant provides 

employment 

and income situation. 



documented evidence of need to improve 
(11-10, Al) 



■ Proposed vocational education effort is tied into larger economic 
development effort. (11-11, Bl) 



■ Reliance on statements of longstanding need and intent to hire indicate 
realistic objectives. (11-11, Bl) 



On the next page you will find a poor comment followed by a critique and a good comment. 
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MODULE Ills Writing Comments 



Poor Comment 

The applicant plans to reduce high unemployment rates and low income in 
the tribe through vocational education. Four areas are targeted for project 
efforts (II- 11 to 11-12). Objectives are realistic. 

Critique 

This comment is too general and offers only a description of the applicant's 
plans. The first sentence does not identify how need for the project is 
documented by the applicant, nor does it indicate how/how well vocational 
education will meet that need. The four target areas of the project are not 
identified and no information is given to indicate how well they respond to 
the identified need. Although a page reference is provided, program staff 
should not have to refer to the application for such basic information; page 
references should indicate particularly significant aspects of the application. 
Finally, the comment fails to specify the nature and number of proposed 
objectives or to provide support for their characterization as "realistic." 



Good Comment 

The applicant proposes to reduce tribal unemployment (currently at 69 
percent) and improve tribal income (currently below the Federal poverty 
level), thereby decreasing tribal dependence on Federal and State economic 
aid (11-10, Al). The proposed project will promote jobs and businesses on 
the reservation in accordance with the tribe's Comprehensive Development 
Plan (CDP). Construction of a new Interstate highway near the reservation 
makes new economic growth possible (11-10, A2), and the CDP lays the 
groundwork for future printing and electronics assembly plants, housing, and 
shopping areas (Attachment A). The proposed project will train tribe 
members for jobs in printing, electronics assembly, and carpentry. These are 
and will continue to be "demand" occupations undersupplied in the Cimarron 
Metropolitan Area and were selected for the vocational education effort 
based on statements from local employers concerning longstanding need and 
intent to hire (11-11, Bl). A fourth area, tourism, is also targeted by the plan 
to provide off-reservation jobs in the newly developing Foothills Recreation 
Area (11-12, B2). Such jobs are expected to integrate the tribe with the CMA 
economic base. The four areas of effort are realistic avenues to tribal self- 
sufficiency. 




m-5 

68 



MODULE III: Writing Comments 



EXERCISE: EXTENT OF NEED FOR THE PROJECT 

Directions: Use the selected weaknesses listed below to write a good comment. Document 
your comment with direct evidence from the passage provided on II-IO to 11-12. 



Selected Weaknesses 

■ The application lacks specific data indicating the availability of jobs, 
(11-11 to 11-12, Bl-2) 

■ Little detail is provided on how the project will move trainees 
into the work environment. (II- 11 to 11-12) 



Comment 



Compare your comment to the sample comment provided on II-8. The sample comment is 
followed by a list of its strong points. 
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Good Comment 

Although the applicant provides a general description of the growing 
employment opportunities in the area, specific details about the availability 
of jobs in the area are not given (II- 11 to 11-12, Bl-2). Therefore, it is 
difficult to assess whether the project's vocational education activities target 
marketable skills necessary to alleviate the tribe's employment needs (11-10, 
Al). In addition, the applicant does not explain how the project will place 
trainees into the work environment (11-11 to 11-12). The lack of any such 
placement plan will detract from the project's ultimate success. 



Why this is a good comment: 

■ Uses proper grammar and spelling. 

■ Presents weaknesses clearly and concisely. 

■ Provides page and section citations. 

■ Explains the significance of each weakness. 
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PREVIEW EXERCISES 

Directions: This example provides examples of good and poor comments for the Plan of 
Operation criterion. Review guidelines for this criterion are given below. The following 
page provides a list of selected weaknesses that form the basis of the exercise's sample 
comments. These weaknesses are drawn from the application passage provided on 11-16 to 
11-19. You may want to reread this passage to refresh your memoiy. The list of selected 
weaknesses is followed by a poor comment, that is critiqued to identify its flaws. A good 
comment follows the critique. Read these comments and study the critique to prepare 
yourself for writing comments. 



Criterion: Plan of Operation 



Review Guidelines 

1) Do the project objectives serve the purposes of the authorizing statute? 

2) How well is the project designed? 

Are project objectives consistent with stated needs? 

Are project activities consistent with project objectives? 

Are project objectives measurable? 

3) How will the applicant use its resources and personnel to achieve 
each objective? 

4) Has the applicant developed an effective management plan that will 
ensure proper and efficient administration of the project? 

5) Do project milestones represent a logical progression of times and 
tasks? 

6) Does the applicant propose a realistic time schedule for accomplishing 
objectives? 

7) Will the proposed activities accomplish the project's objectives 
successfully? 

8) Are the educational approaches planned based on sound research that 
indicates they will be successful tor the population to be served? 

9) Does the project have clearly developed provisions for providing equal 
access to eligible participants who are members of traditionally 
underrepresented groups (racial or ethnic minorities, women, handicapped 
persons, elderly persons)? 
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Selected Weaknesses 

■ Goal 4 is not measurable as stated. (11-16, A4) 

■ There seems to be no plan to assess the basic skills of the students and to 
tailor instruction to their skill levels. (11-17 to 11-18, C-D) 

■ The time line is very general. Although some details about the number 
of classroom hours, etc., are given in the plan of operation narrative, 

it is not clear how all activities will fit together ana be managed effectively. 
(H-19, F) 

On the next page you will find a poor comment followed by a critique and a good comment. 
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Poor Comment 

One goal of the project is not measurable. In addition, the plan of operation 
makes no provision for assessing the basic skill level of vocational students. 
The time line for accomplishing objectives is also very general. 

Critique 

This comment does not offer any specific information or documentation. 
The first sentence gives no clue as to what the goal is or why it is not 
measurable. As it stands, it is a statement of opinion rather than an objective 
analysis. The second sentence makes a valid point, but does not provide any 
context for understanding the point. This point should be tied to a 
description of proposed activities and its ramifications in terms of project 
success should be made clear. Finally, the comment does not indicate what 
essential information is missing from the time line and why the lack of that 
information detracts from the proposal. 



Good Comment 

The plan of operation proposes four goals. The first three-decreasing 
unemployment, job placement, and increasing off-reservation employment- 
are measurable. The fourth goal, however— increasing Emanon self-esteem- 
is not measurable as stated and the plan of operation suggests no procedure 
for assessing and monitoring the success of this goal (11-16, A). Since the 
overall objective of the project is to increase Emanon economic 
independence, some procedure for measuring self-esteem would be essential 
for determining whether the project promotes attitudes of self-sufficiency. A 
second weakness is the absence of any plan to assess students' basic skill 
levels (11-17, C). Since the project concentrates its efforts in vocational 
education, its success will depend on how well students are trained. If 
instruction is not conducted at a skill level appropriate to the students, the 
trainees will not be equipped to compete for jobs. Third, the time line 
proposed for accomplishing project activities is not comprehensive or 
detailed. The number of classroom hours to be offered is noted (11-17, C), 
but the hours are not justified by any documentation indicating that they are 
sufficient for the objectives involved. Courses are scheduled to finish by the 
time job opportunities will be available (11-18 to 11-19, F), but no detailed 
master schedule is provided for all activities-recruitment, startup, teacher 
training, instruction, monitoring, evaluation, etc. A comprehensive 
management plan is not evident. 
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EXERCISE: PLAN OF OPERATION 



Directions: Use the selected strengths listed below to write a good comment. Document 
your comment with direct evidence from the passage provided on 11-16 to 11-19. 



Selected Strengths 



■ A close relationship between training, availability of jobs in the area, 
and employer needs is proposed. (11-16, Bl-4; 11-17, C2,;II-18, C4) 

■ The first three goals are measurable. (11-16, Al-3) 

■ The first four objectives are detailed and documented. (11-16, Bl-4) 



Comment 



Compare your comment to the sample comment provided on III-14. The sample comment 
is followed by a list of its strong points. 
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Good Comment 

The applicant proposes three measurable goals: to decrease the tribal unemployment rate, 
to place 85 percent of project graduates into jobs, and to increase the number of tribe 
members in off-reservation jobs (11-16, Al-3). The first four project objectives detail the 
number of Emanons to be trained in each of the project's vocational education areas and 
provide job projections (Attachment C(2)) to support the training objectives (11-16, B 1-4). 
According to the documentation provided, the objectives seem adequate to accomplish the 
project's goals. In addition, the plan of operation is designed to ensure that training 
activities are tailored to the availability of jobs in the area (11-16, Bl-4; 11-17, C2, C4). In 
two of the vocational education areas targeted by the project-electronics assembly and 
tourism-the applicant has already arrangedfor trainee placement (11-17, C2, C4). 



Why this is a good comment: 

■ Uses proper grammar and spelling. 

■ Identifies strengths clearly and concisely. 

■ Provides page and section references. 

■ Identifies the three measurable goals. 

■ Notes the specificity of project objectives and cites supporting 
documentation. 

■ Relates project objectives to project goals to emphasize their 
strength. 

■ Gives specific examples of close relationship between project 
training and availability of jobs. 
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ADDITIONAL EXERCISES 

Directions: This exercise provides examples of good comments for the four remaining 
selection criteria: Quality of Key Personnel, Budget and Cost Effectiveness, Evaluation 
Plan, and Adequacy of Resources. Review guidelines for each criterion are given first, 
followed by two selected strengths and a good comment, then two selected weaknesses and 
a good comment. Read these comments carefully, referring to the indicated passages in 
Module II, to sharpen your sense of what constitutes a good comment. If you want 
additional practice, tiy writing comments of your own before reading the comments 
provided in the text. 



Criterion: Quality of Key Personnel (11-24) 



Review Guidelines 

1) Do the job descriptions adequately reflect skills needed to make the 
project work? 

2) Are the duties of personnel clearly defined? 

3) What relevant qualifications do the proposed personnel possess, 
especially the Project Director? (Focus on their experience and 
training in fields related to the objectives of the project, though 
other information may be considered.) 

4) Will proposed personnel need to be trained for the project? 

5) How much time will the proposed personnel actually devote to the 
project? 

6) To what extent does the applicant encourage employment applications 

from members of traditionally underrepresented groups (ethnic or racial 
minorities, women, handicapped persons, elderly persons)? 



Selected Strengths 



■ Project Director has prior, relevant experience supervising a vocational 
education program. (11-24, A) 

■ Resumes are provided. (Attachment E) 
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Good Comment 

The Project Director has successfully managed two Federal grants for the 
tribe before: one in adult vocational education, the other in adult literacy 
(Attachment D). The Project Director also has a degree in education as well 
as over 9 years' experience working with tribal and other educational 
programs. Her resume (Attachment E) provides a detailed summary of her 
previous activities and indicates that she is seasoned in handling fiscal and 
management responsibilities and has designed vocational programs that 
included job placement components. Resumes are also provided for the 
project secretary and the vocational counselor (Attachment E). 



Selected Weaknesses 

■ Job descriptions are not provided. Hard to judge if personnel have the 
skills necessary for their proposed roles. (H-24) 

■ Other personnel, such as instructors, are not listed. Applicant does not 
explain how these personnel will be selected or what methods will be 
used to ensure that chosen instructors are sensitive to the special needs 
of the Indian population they will teach. (11-24) 



Good Comment 

Although resumes are provided for the three major personnel, the 
application does not provide job descriptions-other than a title--for their 
positions. Without knowing exactly what their positions entail, it is difficult 
to ascertain whether these personnel possess all the skills and experience 
they will need to manage the project. The application also fails to identify 
proposed instructors for the vocational education and job readiness classes. 
Five instructors will be needed, but no description of how they will be 
selected or what criteria will be used for selection is provided. Given the 
project's dependence on vocational education for its success, the lack of 
planning for providing these personnel is a weakness in the project design. 
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Criterion: Budget and Cost Effectiveness (11-28 to 11-30) 



Review Guidelines 

1) Is the budget adequate to support the project's proposed activities? 

2) Are overall project costs reasonable in relation to project objectives? 

3) How much of the project's total cost is devoted to administrative costs? 

4) Are budget items sufficiently justified? 

5) Is the budget padded? 



Selected Strengths 

■ Applicant provides comprehensive cost assessment for all project years. 
(11-28 to 11-30) 

■ Salaries for Project Director and Secretary seem reasonable. 

(11-28, Al) 



Good Comment 

The budget covers all costs, both direct and indirect, for the three years of 
the project, including startup costs. The Project Director's salary is budgeted 
at $33,000, which seems very reasonable considering that she will be 
responsible for managing all aspects of the project (11-18). The Secretary's 
salary is also very Tow ($10,000), although it is not clear how much 
administrative support she will be required to provide. 



Selected Weaknesses 



■ Budget offers no narrative justification. For example, are instructor 
hourly rates at prevailing area wages? (11-28, A3) 

■ Cost of $5,000 for job readiness consultant seems high. (11-29, G) 
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Good Comment 

A detailed breakdown of costs is provided for each project year, but there is 
no narrative justification for the costs that are listed. The application should 
indicate how these costs were established and explain why each is reasonable. 
In particular, the cost of $5,000 for the job readiness consultant seems high 
(11-29, G). Ten workshops are noted, which makes the cost of each workshop 
$500. It is not clear whether these workshops last one hour or one week. 
The application should explain how this cost was determined and provide 
data to show that the price is consistent with current rates of instruction in 
the area. 
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Criterion: Evaluation Plan (11-34) 



Review Guidelines 

1) Are the proposed methods of evaluation appropriate to the project? 

2) Will the proposed evaluation be objective? 

3) Will the proposed evaluation methods measure the effectiveness of 
project activities in meeting project objectives? 

4) Will the evaluation plan produce valid and reliable data concerning the 
accomplishment of project objectives? 

5) Does the evaluation plan measure the project's effect on the project 
audience? 



Selected Strengths 

■ Evaluation plan includes feedback from employers as to effectiveness 
of activities. (11-34, D) 

■ Followup evaluation will measure successful accomplishment of 
objectives. (11-34, D) 



Good Comment 

The project will use feedback from employers who hire project trainees to 
evaluate the success of the vocational education activities (11-34, D). Details 
about how feedback will be obtained are not given, but such evaluation does 
not need to be quantifiable. Informal commentary by employers will alert 
project personnel to insufficiencies in the training program as well as any 
shifts in job opportunities. Project activities can be modified accordingly. 
The project will also conduct followup evaluation of trainees' employment 
status and success (11-34, D). Followup will indicate whether and how well 
the project's objectives are being fulfilled. 
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Selected Weaknesses 

■ Evaluation instruments are not detailed. Who will prepare them? Need 
more inf ormation on the instruments and methods that will be used to 
measure student progress in the vocational education areas of the 
project. (11-34, A, C, D) 

■ A majority of the evaluation is ongoing and relies heavily on the 
subjective assessment of project staff. (11-34, A-C) 



Good Comment 

Although the project intends to use tests to evaluate student progress, these 
instruments are not described (11-34, A). The usefulness of these evaluation 
tests will depend on their exact nature. For example, if the instructors 
prepare their own tests, the results would be more likely to reveal the 
students' real progress than if other standardized tests are used that measure 
the students against others nationwide. On the other hand, standardized 
tests might be better for evaluating students' abilities to compete for and 
retain jobs after they have been trained. The application should be more 
specific about the purpose behind evaluations of student progress and 
indicate exactly how the evaluation will be performed. Similarly, most 
evaluations of the vocational education component are vaguely ongoing and 
no schedule other than the label "regular" is identified for the evaluation 
process. This evaluation is also of questionable utility since it relies almost 
exclusively on the subjective assessments of project instructors and staff. 
Some form of objective, ongoing evaluation should be included for balance. 
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Criterion: Adequacy of Resources (11-38) 



Review Guidelines 

1) Are the proposed facilities adequate for project purposes? 

2) Is the proposed equipment adequate for project purposes? 

3) Does the applicant have access to special sources of experience or 
expertise? 



Selected Strengths 

■ Facilities are within easy commuting distance and equipped for 
handicapped persons. (11-38, B) 

■ Employer that will hire electronics assembly trainees will provide 
the instructor. (11-38, C) 



Good Comment 

The Tribal Hall is adequate to house all of the necessary classrooms, with the 
exception of the carpentry class (11-38, B). The Tribal Hall is located on the 
reservation and will be easily accessible to students. The Tribal Hall is 
already equipped to permit access of handicapped students. Some printing 
classes will be held at the local Technical College, which is also equipped for 
the handicapped. The project also has a sigmficant outside resource in 
Foster-Johnson Electromcs, Inc. --the employer that will hire the electronics 
assembly trainees-which will provide the instructor for the electronics 
assembly course (11-38, C). 



Selected Weaknesses 



■ No detail provided on the physical suitability of facilities for work or 
training. (11-38, B-C) 

■ Facility for carpentry classes does not yet exist. (11-38, B) 



Good Comment 

Although the application proposes facilities to house the various vocational 
education classes, the physical suitability of these facilities is not described. 
Previous passages in the application indicate that the technical portions of 
the printing course will be conducted at the facilities of the Cimarron 
Technical College. The projected carpentry facility is designed for carpentry 
and should be adequate, but since the facility does not yet exist, its suitability 
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and even its existence by the time of project startup are questionable. Job 
readiness classes should not require any special classroom features. 
However, electronics assembly and tourism classes might require more than 
just a classroom. The application should identify what is necessary to each 
class and indicate how these needs will be met. 
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C. Scoring 



The numerical scores you assign to an application's response to the selection 
criteria must be consistent with the comments you write. Comments and 
scores should reflect the same overall assessment. You should never attempt 
to mitigate a negative comment with a positive score, or vice versa. 

Comments indicate whether the application's response to the selection 
criteria is poor, adequate, or good; scores indicate how poor, adequate, or 
good. If 10 points are possible, 0-2 points is poor, 5-7 points is adequate, and 
8-9 points is superior. Four points will indicate a response that is merely 
weak, whereas 8 points will indicate a response that is above average. 
Whatever amount of total points is possible, use the midpoint of the scale as 
adequate and choose your scores accordingly. Do not hesitate to use the full 
range of points. It is perfectly acceptable to assign a score of 10 or 1, for 
example. Your guiding rule should be consistency in rating. 

Always go back and check your scores to make sure that you have written 
them correctly and used the appropriate point scale. You should also 
doublecheck the scores on the summary page of the Technical Review Form 
to make sure that they match the scores listed under each selection criterion 
and that the final total has been computed without error. 

You may want to use the following table as a guide when assigning points: 



Total 


Poor 


Weak 


Adequate 


Superior 


Outstanding 


25 


0-8 


9-12 


13-19 


20-23 


24-25 


20 


0-6 


7-9 


10-15 


16-18 


19-20 


15 


0-4 


5-7 


8-11 


12-13 


14-15 


10 


0-2 


3-4 


5-7 


8-9 


10 


5 


0-1 


2 


3 


4 


5 
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DIRECTIONS FOR SCORING EXERCISE 1 



The following exercise is designed to give you practice in assigning numerical scores based 
on selected strengths and weaknesses. Read through the list of strengths and weaknesses 
and assign an appropriate score in the space provided at the bottom of the page. 
Remember to distinguish between significant and less significant strengths and 
weaknesses when assigning your score. 



Turn to the next page for Exercise 1. 
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EXERCISE 1: EXTENT OF NEED FOR THE PROJECT 
Strengths Weaknesses 



■ Applicant provides 
documented evidence of need 
to improye employment and 
income situation. 

■ The goal of the proposed 
project is clearly specified. 

■ The applicant details how the 
economic plan will improve 
the tribal situation. 

■ The project hopes to make 
future government assistance 
unnecessary. 

■ Proposed vocational education 
effort is tied into larger 
economic development effort. 

■ Education program plans are 
not focused narrowly on 
reservation jobs, but address 
the larger need to equip 
Indians to work in joos outside 
the reservation. 

■ Applicant's proposal responds 
to needs clearly documented 
by outside institutions. 

■ Finding off-reservation jobs 
for Emanons is clearly 
possible. 

■ Reliance on statements of 
longstanding need and intent 
to hire indicates realistic 
objectives. 

■ Applicant identifies needs that 
are unique to the target group, 
although these needs are not 
very specific or well 
documented. 



Assessment does not specify the kind and 
number of jobs to be provided by the 
project. 



It is unclear what skills the target group 
lacks that make them unable to find work. 

The application lacks specific data 
indicating the availability of jobs. 



Little detail is provided on how the 
project will move trainees into the work 
environment. 

Applicant needs to provide statistics to 
support claim of high dropout rate for 
those employed outside the reservation. 



How many points would you give? 

Maximum 25 points Awarded 

Compare your score to the score given on the next page and read the discussion of why it 
was chosen. 



MODULE III: Scoring 



This section should receive a score of approximately 18 points. 

DISCUSSION 

The applicant specifies the need for the project and provides documented evidence of that 
need. TTie applicant also indicates that the project is part of a larger economic 
development effort and offers some proof that the project will be able to meet the tribe's 
need successfully by providing job skills sought after by focal employers. 

The applicant does not, however, offer enough specific data on how many jobs will be 
provided or on the availability of employment. Similarly, no data on tribal members' lack 
of skills is given, making it difficult to determine whether the vocational education effort is 
appropriate for them. The applicant also assumes that trainees will find employment 
without explaining how placement is to be conducted. 

Thus, although the application adequately describes an evident need and a plan to resolve 
that need, it needs to detail how the two are related. The weaknesses suggest that the 
applicant has not analyzed the situation in depth. Therefore, the response is adequate but 
not superior. 
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DIRECTIONS FOR SCORING EXERCISE 2 



The exercise on the next page provides another list of selected strengths and weaknesses 
for a different selection criteria. Read the list and assign an appropriate score in the 
space provided at the bottom of the page. Remember to distinguish between significant 
and less significant strengths and weaknesses when assigning your score. 



Turn to the next page for Exercise 2. 
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EXERCISE 2: PLAN 
Strengths 



■ Evidence demonstrates that 
the CDP has been found 
viable by other grant-making 
organizations. 

■ Close relationship between 
training and availability of jobs 
in the area, employer needs, is 
proposed. 



■ First three goals are 
measurable. 

■ Exact number of persons to be 
trained is given. 

■ The first four objectives are 
detailed and documented. 

■ Training objectives relate to 
overall economic plans of the 
tribe. 

■ Resources are detailed when 
necessary. 

■ Responsibilities of key 
personnel are outlined. 



DF OPERATION 

Weaknesses 

Not clear how "dropout" rate for off- 
reservation actiyities is to be cured by 
vocational training. How will "dropout" 
attitude be attacked specifically? 

First goal seems very optimistic — can 
proposed project effect such a drastic 
reduction m unemployment given the 
diverse factors that lead to unemployment 
in this population? 

Goal 4 is not measurable as stated. 

Specific objectives for the job readiness 
component are not clear. 

There seems to be no plan to assess the 
basic skills of the students and to tailor 
instruction to their skill levels. 

The educational methodology to be used 
with adults who have low skill levels and 
low self-esteem is not clear. 

Recruitment procedures are not specific. 

Selection procedures are not described. 

Burden of Project Director is too heavy. 

Claim that accounting and audit system 
will be satisfactory by Federal standards 
should be documented. The nature and 
extent of oversight should be clarified. 

The time line is very general. Although 
some details about the number of 
classroom hours, etc., are given in the 
lan of operation narrative, it is not clear 
ow all activities will fit together and be 
managed effectively. 

Application fails to detail startup 
activities. 



How many points would you give? 
Maximum 20 points Awarded _ 



Compare your score to the score given on III-30 and read the discussion of why it was 
chosen. 
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This section should receive a score of approximately 9 points. 

DISCUSSION 

Most of the applicant's goals and objectives seem well designed to meet the outlined need. 
Four of the objectives are detailed and related to the tribe's economic plan. Resources for 
conducting training are delineated and the responsibilities of key project personnel are 
outlined. 

On the other hand, the fourth goal is not measurable and the fifth objective is unclear. 
Little information is provided on educational methodology, recruitment, and selection of 
students. The activities to be performed are only vaguely described and no schedule of 
activities is provided. Fiscal procedures are not detailed and the responsibilities of the 
Project Director are too heavy. 

The applicant thus presents only a very general plan of operation. A comprehensive and 
coherent management plan is not provided, which suggests that the applicant is not 
adequately prepared to deal with the practical aspects ofimplementing the project, much 
less any unforeseen complications. Therefore, the response is not adequate, despite what 
appear to be reasonable goals and objectives. 
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DIRECTIONS FOR ADDITIONAL SCORING EXERCISES 

The next four pages provide lists of strengths and weaknesses for the following selection 
criteria: Quality of Key Personnel, Budget and Cost Effectiveness, Evaluation Plan, 

Adequacy of Resources. Practice scoring by reading these lists and assigning what you feel 
is an appropriate score in the space provided. You can then compare your scores to those 
listed in the answer key on 11-36. 



Turn to the next page for additional exercises. 
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EXERCISE 3: QUALITY OF KEY PERSONNEL 
Streng ths Weaknesses 



■ Project Director has prior, 
relevant experience 
supervising a vocational 
education program. 



■ Secretary is a graduate of one 
of Tribe s previous vocational 
education programs. 

■ Counselor has previous 
experience in educational 
counseling. 



Amount of time personnel will devote to 
the project is not given. 

Job descriptions are not provided. Hard 
to judge if personnel have the skills 
necessary to their proposed roles. 

Other personnel, such as instructors* are 
not listed. Applicant does not explain 
how these personnel will be selected or 
what methods will be used to ensure that 
chosen instructors are sensitive to the 
special needs of the Indian population 
they will teach. 



■ Resumes are provided. 



How many points would you give? 



Maximum 15 points Awarded 
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EXERCISE 4: BUDGET AND COST EFFECTIVENESS 



Strengths 

■ Applicant provides 
comprehensive cost 
assessment for all project 
years. 

■ Salaries for Project Director 
and Secretary seem 
reasonable. 



Weaknesses 



Budget offers no narrative justification. 

Fringe benefits are not itemized or 
justified. 

Is the counselor's presence at the 
conference necessary? 

Do carpentry students require their own 
set of tools? Couldn't these be shared by 
all the classes? 

Cost of $5,000 for job readiness 
consultant seems nigh. 

No effort seems to have been made to 
obtain contributions from the Tribe or 
businesses. 

Project intends to train 75 students, which 
makes the project cost per student around 
$4,900 year. Amount seems high. 



How many points would you give? 



Maximum 10 points Awarded 
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EXERCISE 5: EVALUATION PLAN 



Streng ths 

■ Evaluation procedures include 
instructor feedback. 

■ Evaluation procedures include 
student feedback. 

■ Student survey form was used 
with prior success. 



■ Evaluation plan includes 

feedback from employers as to 
effectiveness of activities. 



■ Followup evaluation will 
measure successful 
accomplishment of objectives. 

■ An evaluation instrument is 
included to measure the effect 
of the human development 
component of the job 
readiness workshops. 



Weaknesses 

Evaluation instruments are not detailed. 
Who will prepare them? Will they be 
standardized tests or tests prepared by the 
instructor? 



No indication of how students will be 
monitored or how counseling will be 
coordinated. 

Need more information on the 
instruments and methods that will be used 
to measure student progress in the 
vocational education areas of the project. 

The method and frequency of instructor 
feedback is not stated. 



Applicant should include a copy of the 
survey form for review. 

It is not clear how survey data will be 
used to improve instruction. 

The method and frequency of feedback 
from employers is not clear. 

Evaluation plan does not explain how 
placement data will be reported. 

A majority of the evaluation is ongoing 
and relies heavily on the subjective 
assessment of project staff. 

Plan does not indicate how program 
improvement needs will be identified. 

No provision is made for outside 
evaluation to assure the objectivity of 
internal evaluation. 

The reliability and validity of evaluation 
data is not discussed. 

The budget includes no provision for 
purchasing or scoring evaluation 
instruments. 



How many points would you give? 



Maximum 15 points Awarded 
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EXERCISE 6: ADEQUACY OF RESOURCES 

Streng ths Weaknesses 

d Facilities are .within easy ■ No detail provided on the physical 

commuting distance and suitability of facilities for work or training, 

equipped For handicapped 

persons. ■ Facility for carpentry classes does not yet 

exist. 

b Equipment and facilities are 
protected and secure. 

■ Employer that will hire, 
electronic assembly trainees 
will provide the curriculum. 



a Available equipment seems 
adequate for training printers. 



How many points would you give? 



Maximum 10 points Awarded 
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ANSWER KEY FOR EXERCISES 3-6 


Exercise 3: Quality of Key Personnel 


7 points 


Exercise 4: Budget and Cost Effectiveness 


6 points 


Exercise 5: Evaluation Plan 


5 points 


Exercise 6: Adequacy of Resources 


7 points 
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D. Team Skills 



You may be asked to participate in a panel discussion, which will supplement 
your individual review. The purpose of panel discussion is not to present just 
your own assessment, but to provide a constructive airing of all views that will 
ensure a thorough assessment of an application. You should approach panel 
discussion as a team effort to clarify individual evaluations, 
misunderstandings, and questions. You should not attempt to convince other 
panel members that their assessments are wrong, although panel members 
are allowed to alter their original evaluations if they feel, based on the panel 
discussion, that these are incorrect. You are responsible for presenting your 
views objectively and considering the views of other panel members. 

Active Listening 

As a panel member, you are expected to participate in discussion. But the 
quality of your input will depend to a large extent on your listening skills. 
Just as you must actively read applications in order to provide a valuable 
assessment, you must actively listen to the points raised during panel 
discussion in order to provide useful responses. You cannot rebut an 
erroneous statement, reinforce a particular point, or supply significant 
information that has been overlooked if you are not paying attention to what 
is being said. 

Your listening skills must be comprehensive. Comprehensive listening 
involves concentrating on understanding what is said. The active listener will 
distinguish between factual information, information based on 
interpretations of data, and generally unfounded opinions or bias. As a 
panelist, you must be alert to statements that are not based on information 
included in the application but on professional judgment and give them their 
appropriate weight. If you sense an unwarranted bias, you must respond by 
tactfully pointing out the unsupported bias. Your concern and responsibility 
is to ensure that the discussion is objective and presents the whole picture. 
Because you have already reviewed the applications under discussion, you 
must listen critically to the statements of other panel members rather than 
accepting or rejecting them out of hand because they happen to support or 
contradict your own impressions. If your assessment of an application is not 
shared by other panel members, you are obligated both to present your own 
view and to reconsider your view in light of what other panel members 
present. 

As with active reading, active listening may be impaired by certain internal or 
external distractions: noise, physical discomfort (hunger, pain, fatigue), 

emotional states (boredom, anger, preoccupation with personal problems), 
and simultaneous physical activity (tapping a pencil, playing with a paper 
clip). You should be alert to these distractions and make an effort to 
minimize their effect on your listening ability. 
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One critical skill in active listening is paraphrasing. This involves listening 
carefully to what someone else has said and then translating this into your 
own words and repeating it before you add your own thoughts. As an 
example, consider tne following: 

Panelist A: "I gave a high score on the evaluation criterion, because they plan 
to use a variety of instruments to get feedback from students and 
employers. A lot of projects never get employer feedback." 

Panelist B: "If I understand you, you believe that the evaluation plan is 

strong because it will gather data from students and employers. 
While that is a strength, I am concerned that these instruments 
were not described or provided in the application. That lack of 
specificity is a significant weakness to me." 

Paraphrasing is a useful skill for three reasons. First, in order to paraphrase, 
you have to concentrate on what the other person is saying. That 
concentration makes sure you truly "hear" another's arguments, instead of 
formulating what you are going to say while someone else is talking. Second, 
if your paraphrasing is inaccurate, others will be able to correct you so that 
you do not proceed with the wrong impression of what took place. And third, 
when you paraphrase, you show concern for other people and their points of 
view. When others see you trying to restate their points, they may do the 
same. This creates a good team atmosphere that is conducive to the panel 
process. 



LISTENING TIPS 

1) Focus on understanding what is being said. 

2) Paraphrase what the previous panelist said before stating 
your views. 

3) Make a critical evaluation of what is being said. 

4) Be receptive to opposing viewpoints. 

5) Reduce noise and physical discomfort. 

6) Put your emotions aside. 




m-38 

98 



MODULE III: Team Skills 



Team-Building Tips 

There are several things you can do to help keep your panel working 
smoothly and efficiently: 

1) Suggest that your panel decide on an agenda and approach at the start 
so that it will manage its work and time better. 

2) Pay attention to time and bring up the agenda if the panel gets off track 
or falls behind. 

3) Ask others to explain their positions. 

4) Present your own ideas. 

5) Summarize the panel's progress periodically, noting points still to 

be resolved concerning a specific application as well as the number of 
applications still to be discussed. 

6) Participate actively in the discussion, using supporting evidence for 
your points. 

7) Answer other panelists' questions and challenges seriously and 
diplomatically. 

Remember to be patient, courteous, and tactful toward other panel 
members. Think of the panel as a team that must pull together. 




III-39 



MODULE Ills Team Skills 



EXERCISE: TEAM BUILDING 

Directions: The following exercise presents situations that might arise during panel 
discussion. Read the description of each situation carefully and consider how best to 
handle it. Note your suggestions in the space provided. 



SITUATION 1 

There are three people on your panel, but one of them has not participated 
in the discussion at all. You have asked the silent panelist for his opinion 
twice, but the panelist merely asserted agreement with what was being said. 

How could you involve the silent panelist? 



SITUATION 2 

A member of your panel frequently interrupts other panelists with long, 
seemingly unfounded comments that do not address the issues under 
discussion. Another panel member reacts to one such interruption by 
remarking that there's no use in even trying to make a point. 

How could you improve this unproductive situation? 



MODULE III: Team Skills 



SITUATION 3 

A fellow panel member refuses to move on to another topic. He seems 
determined to have his position accepted by the other panelists and 
continues to demand and then criticize their differing opinions. 

How could you settle the issue? 



SITUATION 4 

One panel member is very witty and has created an atmosphere of frivolity 
that has the other panel members joking about the application and telling 
marginally relevant anecdotes. 

How could you place the discussion on a more serious footing? 



Turn to the next page for a list of possible approaches to improving these situations. 
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POSSIBLE APPROACHES 



SITUATION 1 

You could ask the silent panelist to open the discussion of the next selection 
criterion. You could also try asking the silent panelist for his/her opinion 
about another panelist's remarks. If the silent panelist merely agrees again, 
question one of the other panelist's points and request clarification from the 
silent panelist. 



SITUATION 2 

This troublesome panelist clearly likes to talk, but doesn't listen carefully to 
the discussion. To smooth over this moment of tension, you could quickly 
pick up the point introduced by the affronted panelist and ask the 
troublesome panelist to clarify the point for your benefit. Or you might 
restate the affronted panelist's point yourself and ask the troublesome 
panelist to comment on it directly. 



SITUATION 3 

If you are very tactful, you could remind the entire panel that panel 
discussion is not meant to produce a consensus of opinion. You could also 
suggest that everyone agree to disagree. Another strategy would be to 
acknowledge the conviction of the stubborn panelist and introduce another 
issue as one you feel strongly about. 



SITUATION 4 

You could broach a serious topic and ask the witty panelist for his/her 
opinion. If the witty panelist responds with another anecdote, you could 
admit to not understanding his/her point and ask for straightforward 
clarification. You could also follow an anecdote with the suggestion that 
"we" should get back to main issue. A reminder that available time for the 
review is limited might be sobering too. 
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Conclusion 



Congratulations! You have now finished the last module of the workbook on 
grant application review. If you read the material and completed some or all 
of the exercises, you should feel comfortable with your role as a reviewer and 
confident of your ability to review grant applications. But before you put the 
workbook away, review your original objectives. 

Do you understand: 

■ The review process and your responsibilities as a reviewer? 

■ The roles of other review personnel? 



Can you: 

■ Read and analyze applications efficiently? 

■ Apply selection criteria? 

■ Write detailed comments? 

■ Score applications correctly? 

■ Serve on a panel? 

If you are unsure about any of the above, go back and review the relevant 
sections of the workbook. When you feel that you have accomplished all the 
above objectives, you will be ready for the challenge and responsibility of 
application review. Then you can put the workbook away-but remember to 
keep it for future reference. 

Thank you for all your effort. 
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Cooperative Agreements: Grants in which the awarding agency has substantial 
involvement in administering project activities and supervising their outcome. 

Discretionary Grants: Grants awarded at the discretion of a Federal administrator, 
subject to conditions specified by legislation. 

EDGAR: The Education Department General Administrative Regulations. A 
handbook that details the regulations (34 CFR; Parts 74-79) governing Department 
of Education activities. 

Grants Officer: A member of the staff of the Grants and Contracts Service office 
who may conduct orientation sessions for reviewers and act as an adviser during 
panel discussion. The Grants Officer can answer questions concerning grants 
management policy, the provisions of EDGAR, conflict of interest, and 
confidentiality and is responsible for the negotiation, award, and administration of 
grants. 

Panel: A group of two or more reviewers convened to discuss applications. A panel 
can be either the regular review method for a group of applications or an adjunct to 
individual reviews. The purpose of a panel is to ensure thorough consideration of 
each application. Panels may include Department of Education officials, who act as 
advisers or facilitators. 

Panel Manager/Chairperson: A Department of Education official responsible for 
panel logistics and facilitation. This official provides information on procedural and 
administrative matters. 

Program: An area of educational emphasis within the Department of Education. 
Programs administer the award of Federal funds to projects that meet legislated 
educational objectives. 

Program Officer: A member of the professional staff of a Department of Education 
program. This official acts as an adviser on program policy during panel reviews 
and may act as Panel Manager. The Program Officer checks completed Technical 
Review Forms for adequacy and accuracy and also uses reviewers' evaluations to 
prepare summary funding recommendations. 

Project: A term denoting the proposed set of activities or scope of work for which 
an application for funding is made to the Department. 

Reviewers: Persons selected by the Department of Education to read and evaluate 
applications for funding. 

Selection Criteria: The evaluation standards that applicants must respond to and 
reviewers must apply in assessing grant applications. Seven general selection 
criteria are listed in EDGAR (Meeting the Purposes of the Authorizing Statute, 
Extent of Need for the Project, Quality of Key Personnel, Budget and Cost 
Effectiveness, Evaluation Plan, Adequacy of Resources), but programs often have 
slightly different or additional criteria that reflect their particular concerns. 

Technical Review Form: The official document that reviewers complete when 
evaluating applications. 
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